Oct 30 00:28:57.307669 ip-10-0-131-130 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 00:28:57.307678 ip-10-0-131-130 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 00:28:57.307685 ip-10-0-131-130 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 00:28:57.307903 ip-10-0-131-130 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 00:29:07.468556 ip-10-0-131-130 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 00:29:07.468577 ip-10-0-131-130 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot dbedf14cfe034ec6b35e8a18cebb67d6 -- Oct 30 00:31:14.652558 ip-10-0-131-130 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:31:15.020811 ip-10-0-131-130 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:31:15.020811 ip-10-0-131-130 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:31:15.020811 ip-10-0-131-130 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:31:15.020811 ip-10-0-131-130 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:31:15.026709 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026605 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.026709 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026705 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.026709 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026710 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.026709 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026714 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026718 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026723 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026726 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026730 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026734 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026737 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026741 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026745 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026748 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026752 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026756 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026760 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026773 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026778 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026782 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026786 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026793 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026798 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.026960 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026803 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026807 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026811 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026815 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026819 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026823 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026827 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026835 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026840 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026845 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026850 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026856 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026860 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026865 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026869 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026873 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026877 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026881 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026885 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.027737 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026889 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026894 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026898 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026902 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026907 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026911 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026915 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026920 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026924 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026929 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026933 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026938 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026943 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026947 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026951 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026955 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026959 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026963 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026967 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026971 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.028556 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026975 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026979 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026988 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026993 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.026997 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027002 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027006 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027010 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027014 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027018 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027022 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027027 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027031 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027035 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027039 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027043 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027047 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027052 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.029404 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027056 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027061 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027069 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027075 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027081 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027086 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027090 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027095 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027099 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027103 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027108 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027112 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027116 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027121 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027125 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027133 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027137 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027142 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027146 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.030349 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027151 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027155 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027159 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027163 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027168 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.027175 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028437 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028452 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028457 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028461 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028465 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028471 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028475 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028479 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028484 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028489 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028494 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028498 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028503 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.030811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028507 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028511 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028515 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028519 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028526 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028533 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028538 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028542 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028549 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028553 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028557 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028561 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028565 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028569 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028573 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028577 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028581 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028585 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028590 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028594 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.031460 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028599 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028603 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028607 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028612 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028616 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028620 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028625 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028629 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028634 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028639 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028643 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028648 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028652 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028656 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028660 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028664 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028668 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028673 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028678 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028682 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.032288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028687 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028692 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028696 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028701 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028705 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028709 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028714 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028718 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028723 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028727 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028731 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028735 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028739 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028744 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028748 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028752 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028756 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028760 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028765 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.033041 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028769 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028773 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028778 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028789 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028795 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028799 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028803 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028808 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028812 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028816 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028821 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028825 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028831 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028835 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028839 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028844 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028848 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028852 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.033721 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028856 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028860 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028864 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028868 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028872 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028878 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028883 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028887 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028891 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028896 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028900 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028904 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028908 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.028912 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029726 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029736 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029740 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029745 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029750 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029754 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.034189 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029759 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029763 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029772 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029776 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029780 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029787 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029791 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029796 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029800 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029804 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029809 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029813 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029818 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029822 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029825 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029830 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029834 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029839 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.034689 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029843 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029847 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029851 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029856 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029860 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029864 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029868 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029874 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029878 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029884 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029888 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029892 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029896 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029901 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029904 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029909 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029913 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029918 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029922 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029928 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.035156 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029933 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029937 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029941 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029945 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029949 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029953 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029958 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029962 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029968 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029972 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029978 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029993 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.029997 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030002 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030006 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030010 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030014 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.035640 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030019 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030022 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030026 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030031 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030034 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030038 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030042 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030046 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030050 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030054 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030057 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030061 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030067 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030081 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030085 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030089 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030093 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030098 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030102 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030106 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.036075 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030110 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030115 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030119 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030122 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030127 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030131 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030135 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030143 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030147 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030151 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030155 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030159 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030167 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030173 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030180 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030185 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030191 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030195 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.036599 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030200 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030204 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.030208 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031123 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031139 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031152 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031160 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031166 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031172 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031179 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031185 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031191 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031196 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031202 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031207 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031212 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031217 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031221 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031226 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031232 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031254 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031261 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031266 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:31:15.037040 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031270 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031276 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031281 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031288 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031292 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031298 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031305 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031311 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031316 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031320 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031325 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031330 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031337 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031342 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031350 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031355 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031359 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031365 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031373 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031378 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031383 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031388 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031393 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031400 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:31:15.037592 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031405 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031410 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031415 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031419 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031424 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031429 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031434 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031439 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031443 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031448 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031455 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031460 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031464 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031469 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031473 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031478 2577 flags.go:64] FLAG: --help="false" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031484 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031489 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031494 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031499 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031505 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031511 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031516 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:31:15.038212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031520 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031525 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031532 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031537 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031542 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031546 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031551 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031556 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031561 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031565 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031569 2577 flags.go:64] FLAG: --lock-file="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031574 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031579 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031584 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031593 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031598 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031602 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031607 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031612 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031617 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031622 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031626 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031634 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031639 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031645 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 00:31:15.038775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031649 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031655 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031661 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031665 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031670 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031675 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031681 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031691 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031697 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031703 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031708 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031713 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031717 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031722 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031726 2577 flags.go:64] FLAG: --port="10250" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031731 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031736 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06cb5623e71db73ee" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031741 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031747 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031752 2577 flags.go:64] FLAG: --register-node="true" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031756 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031763 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031767 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031771 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031776 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031782 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:31:15.039371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031787 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031792 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031797 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031801 2577 flags.go:64] FLAG: --runonce="false" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031806 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031811 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031816 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031821 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031827 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031831 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031836 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031841 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031846 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031851 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031855 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031861 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031866 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031871 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031876 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031884 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031889 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031893 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031899 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031904 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031909 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:31:15.040392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031914 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031918 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031923 2577 flags.go:64] FLAG: --v="2" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031929 2577 flags.go:64] FLAG: --version="false" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031936 2577 flags.go:64] FLAG: --vmodule="" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031942 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.031948 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033818 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033834 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033837 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033841 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033844 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033847 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033850 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033852 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033855 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033858 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033861 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033864 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033866 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033869 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.041428 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033872 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033874 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033877 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033879 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033882 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033884 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033887 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033890 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033892 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033895 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033898 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033901 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033908 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033911 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033913 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033916 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033919 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033921 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033924 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.042383 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033927 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033930 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033933 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033936 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033938 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033941 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033944 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033946 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033949 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033952 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033954 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033957 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033959 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033962 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033964 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033967 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033970 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033972 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033975 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033977 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.043250 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033980 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033983 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033988 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033991 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033994 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033996 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.033999 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034001 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034004 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034006 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034009 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034012 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034015 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034018 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034020 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034023 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034025 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034028 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034030 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.044106 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034033 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034035 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034038 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034041 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034043 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034046 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034049 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034051 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034054 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034057 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034060 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034063 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034066 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034076 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034082 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034087 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034090 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034094 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034096 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034099 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.044708 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034102 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034105 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034107 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034112 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034115 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034118 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034121 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034124 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034126 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034129 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034133 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034137 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034140 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.034169 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034207 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.045527 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034213 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034216 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034220 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034223 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034226 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034230 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034235 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034251 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034255 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034258 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034261 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034264 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034267 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034270 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034273 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034276 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034278 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034281 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034283 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.046104 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034286 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034288 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034293 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034296 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034298 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034301 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034303 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034306 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034308 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034311 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034314 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034317 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034319 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034322 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034324 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034327 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034329 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034332 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034334 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034337 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.046829 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034339 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034342 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034345 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034347 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034350 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034353 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034355 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034358 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034360 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034363 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034365 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034368 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034370 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034373 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034380 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034382 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034385 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034387 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.047317 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034390 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034393 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034395 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034398 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034400 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034403 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034406 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034409 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034411 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034415 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034419 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034424 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034427 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034429 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034432 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034435 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034438 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034440 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034443 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034447 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.047809 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034451 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034454 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034457 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034460 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034463 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034466 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034468 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034473 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034476 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034479 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034482 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034484 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034487 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034490 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034492 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034496 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034501 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034505 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034508 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.048366 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034510 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034513 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034516 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034518 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034521 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034523 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034526 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.034528 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.041924 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.041941 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.041963 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.041970 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042051 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042058 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042063 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042076 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042082 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042086 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042092 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042096 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.048811 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042100 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042105 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042109 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042113 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042117 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042121 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042126 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042130 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042135 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042139 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042143 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042148 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042152 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042156 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042160 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042164 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042168 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042173 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042177 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042181 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.049497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042185 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042191 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042197 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042201 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042206 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042209 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042214 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042219 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042224 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042228 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042233 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042251 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042256 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042259 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042267 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042271 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042275 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042279 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.050420 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042284 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042288 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042292 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042296 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042301 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042305 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042309 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042313 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042318 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042322 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042326 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042330 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042334 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042339 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042346 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042353 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042359 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042363 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042367 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.051194 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042372 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042376 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042380 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042384 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042387 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042391 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042395 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042400 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042406 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042413 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042418 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042422 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042427 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042431 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042435 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042439 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042444 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042448 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.051746 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042457 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042461 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042465 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042469 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042473 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042477 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042482 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042486 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042490 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042494 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042499 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042505 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042510 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042515 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042519 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042523 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042527 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042532 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042536 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.052196 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042540 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042544 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.042587 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042806 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042813 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042818 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042822 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042827 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042831 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042835 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042840 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042844 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042848 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042852 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042857 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042861 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.052714 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042865 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042870 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042874 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042878 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042882 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042887 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042894 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042901 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042907 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042914 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042918 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042922 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042926 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042931 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042935 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042940 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042944 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042949 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042953 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042958 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.053140 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042962 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042967 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042971 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042975 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042980 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042983 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042988 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042992 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.042996 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043000 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043005 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043009 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043013 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043018 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043022 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043026 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043031 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043035 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043039 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.053638 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043048 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043052 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043057 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043061 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043065 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043069 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043078 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043082 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043087 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043091 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043095 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043099 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043103 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043108 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043112 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043116 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043120 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043124 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.054200 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043129 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043133 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043137 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043142 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043146 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043150 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043154 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043158 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043163 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043167 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043171 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043175 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043180 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043186 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043192 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043196 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043201 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043205 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043210 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.054765 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043214 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043218 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043222 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043226 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043230 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043257 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043263 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043266 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043271 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043275 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043279 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043284 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043288 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043293 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043297 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.043337 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:31:15.055211 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043386 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043393 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043397 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043402 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043406 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043410 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043415 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043419 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043423 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043430 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043434 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043438 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043442 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043446 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043452 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043456 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043460 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043464 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043469 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043473 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:31:15.055639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043478 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043487 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043494 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043499 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043504 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043508 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043512 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043516 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043521 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043525 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043529 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043533 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043538 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043542 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043547 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043551 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043555 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043559 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043564 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:31:15.056094 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043568 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043572 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043578 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043582 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043591 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043595 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043600 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043605 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043609 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043613 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043618 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043622 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043627 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043638 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043642 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043646 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:31:15.056587 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043654 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043659 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043663 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043667 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043671 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043675 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043682 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043686 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043690 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043694 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043698 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043703 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043707 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043712 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043716 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043721 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043726 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043730 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043735 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043739 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:31:15.057071 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043743 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043747 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043752 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043756 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043761 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043765 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043769 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043775 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043779 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043785 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043791 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043796 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043801 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043805 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043810 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043815 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043824 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043828 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:31:15.057807 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043833 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043841 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043845 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043851 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043855 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043860 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043864 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:15.043869 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.043940 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.046579 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.047388 2577 server.go:1014] "Starting client certificate rotation" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.047479 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.047512 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.049695 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:31:15.058288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.051374 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:31:15.062217 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.062203 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:31:15.065255 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.065227 2577 log.go:25] "Validated CRI v1 image API" Oct 30 00:31:15.066105 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.066078 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:31:15.070895 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.070878 2577 fs.go:135] Filesystem UUIDs: map[4bb1c90b-3da9-4139-9985-801e8fae73b9:/dev/nvme0n1p3 55befcf6-b936-4fa1-a3a9-2e4f9f2b62dc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 00:31:15.070959 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.070895 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:31:15.077988 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.077771 2577 manager.go:217] Machine: {Timestamp:2026-10-30 00:31:15.076324726 +0000 UTC m=+0.307221240 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103310 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214253f1e7e19900f9aaaaf9787c71 SystemUUID:ec214253-f1e7-e199-00f9-aaaaf9787c71 BootID:dbedf14c-fe03-4ec6-b35e-8a18cebb67d6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:75:f3:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:75:f3:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:5f:71:16:55:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:31:15.078485 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.078473 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:31:15.078571 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.078553 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:31:15.081262 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081227 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:31:15.081391 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081264 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:31:15.081432 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081393 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:31:15.081432 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081401 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:31:15.081481 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081435 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:31:15.081481 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.081453 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:31:15.082204 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.082193 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:31:15.087306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.087295 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:31:15.087419 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.087410 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:31:15.088053 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088044 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:31:15.088086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088056 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:31:15.088086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088066 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:31:15.088086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088074 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:31:15.088086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088082 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:31:15.088401 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.088390 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:31:15.090275 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.090234 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:31:15.094968 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.094956 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:31:15.095666 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095655 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:31:15.095706 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095685 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:31:15.095844 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095835 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:31:15.095873 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095850 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:31:15.095873 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095856 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:31:15.095873 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095862 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:31:15.095873 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095868 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:31:15.095873 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.095874 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:31:15.096422 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096413 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:31:15.096451 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096424 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:31:15.096451 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096437 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:31:15.096451 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096442 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:31:15.096529 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096455 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:31:15.096529 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096464 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:31:15.096997 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096989 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:31:15.097029 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.096999 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:31:15.100575 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.100564 2577 server.go:1263] "Started kubelet" Oct 30 00:31:15.100673 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.100645 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:31:15.100775 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.100733 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:31:15.100825 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.100809 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:31:15.101979 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.101961 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:31:15.102564 ip-10-0-131-130 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:31:15.103940 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.103926 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:31:15.108632 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.108534 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:31:15.108632 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.108626 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:31:15.109103 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109081 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:31:15.109214 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.109200 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.109643 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109626 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:31:15.109736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109630 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:31:15.109736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109657 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:31:15.109736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109721 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:31:15.109736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.109733 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:31:15.110422 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110403 2577 factory.go:153] Registering CRI-O factory Oct 30 00:31:15.110422 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110424 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 00:31:15.110520 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110485 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:31:15.110520 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110495 2577 factory.go:55] Registering systemd factory Oct 30 00:31:15.110520 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110503 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:31:15.110520 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110521 2577 factory.go:103] Registering Raw factory Oct 30 00:31:15.110640 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.110534 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 00:31:15.111540 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.111520 2577 manager.go:319] Starting recovery of all containers Oct 30 00:31:15.121816 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.121803 2577 manager.go:324] Recovery completed Oct 30 00:31:15.125757 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.125742 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.127517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.127499 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.127577 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.127527 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.127577 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.127538 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.128094 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128079 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:31:15.128094 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128090 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.128224 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128092 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:31:15.128224 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128164 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:31:15.128763 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128752 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.128807 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128783 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.128807 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.128796 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.129362 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.129345 2577 policy_none.go:50] "Start" Oct 30 00:31:15.129425 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.129366 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:31:15.129425 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.129380 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:31:15.130486 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.130473 2577 policy_none.go:44] "Start" Oct 30 00:31:15.139619 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.139600 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:31:15.139711 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.139600 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 00:31:15.140400 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.139598 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-130.ec2.internal.18e32749eddf95e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-130.ec2.internal,},FirstTimestamp:2026-10-30 00:31:15.100538336 +0000 UTC m=+0.331434844,LastTimestamp:2026-10-30 00:31:15.100538336 +0000 UTC m=+0.331434844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-130.ec2.internal,}" Oct 30 00:31:15.145355 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.145328 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:31:15.145418 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.145363 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwh2t" Oct 30 00:31:15.153779 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.153761 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwh2t" Oct 30 00:31:15.158613 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.158569 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 00:31:15.158613 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.158577 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:31:15.158613 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.158589 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:31:15.158734 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.158619 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-130.ec2.internal.18e32749ef7b3981 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-130.ec2.internal,},FirstTimestamp:2026-10-30 00:31:15.127515521 +0000 UTC m=+0.358412030,LastTimestamp:2026-10-30 00:31:15.127515521 +0000 UTC m=+0.358412030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-130.ec2.internal,}" Oct 30 00:31:15.167966 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.167941 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:31:15.171075 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171059 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 00:31:15.171167 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.171083 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:31:15.171167 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171090 2577 server.go:87] "Starting device plugin registration server" Oct 30 00:31:15.171355 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171340 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:31:15.171425 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171354 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:31:15.171502 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171479 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:31:15.171618 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171602 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:31:15.171618 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.171616 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:31:15.171934 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.171913 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:31:15.172025 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.171974 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.245179 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.245149 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:31:15.246339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.246320 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:31:15.246410 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.246343 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:31:15.246410 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.246363 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:31:15.246410 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.246401 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:31:15.259543 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.259527 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:31:15.271612 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.271564 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.272557 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.272540 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.272631 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.272569 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.272631 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.272579 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.272631 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.272602 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.290306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.290290 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.290422 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.290310 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-130.ec2.internal\": node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.324854 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.324834 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.346534 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.346505 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal"] Oct 30 00:31:15.346586 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.346573 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.348839 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.348821 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.348923 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.348850 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.348923 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.348861 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.349996 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.349984 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.350154 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.350194 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350167 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.350697 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350683 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.350762 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350706 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.350762 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350728 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.350762 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350736 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.350868 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350711 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.350868 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.350793 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.351799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.351785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.351846 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.351808 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:15.352473 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.352457 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:15.352558 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.352484 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:15.352558 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.352493 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:15.371559 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.371531 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-130.ec2.internal\" not found" node="ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.374764 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.374748 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-130.ec2.internal\" not found" node="ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.411973 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.411954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.412039 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.411976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.412039 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.411991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.425074 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.425054 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.512957 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.512937 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.513026 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.512962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.513026 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.512997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.513026 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.513014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.513122 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.513047 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.513122 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.513075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.525186 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.525146 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.626271 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.626251 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.674389 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.674373 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.677977 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:15.677840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:15.726884 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.726860 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.827379 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.827306 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:15.927869 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:15.927848 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:16.028440 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:16.028417 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:16.047581 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:16.047560 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:31:16.047727 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:16.047696 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:31:17.347005 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:17.346978 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:17.358295 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:17.358139 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b9cc45a336a6ed23d0ff1b116a4584.slice/crio-ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71 WatchSource:0}: Error finding container ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71: Status 404 returned error can't find the container with id ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71 Oct 30 00:31:17.363667 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.363645 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:31:17.367435 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:17.367413 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b21b9005987db40e0d6504b950f5deb.slice/crio-488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a WatchSource:0}: Error finding container 488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a: Status 404 returned error can't find the container with id 488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a Oct 30 00:31:17.375347 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.375322 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:16 +0000 UTC" deadline="2028-06-01 21:08:58.879856799 +0000 UTC" Oct 30 00:31:17.375347 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.375345 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13940h37m41.504515192s" Oct 30 00:31:17.413145 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.413121 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:31:17.447054 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:17.447027 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:17.467502 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.467458 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerStarted","Data":"488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a"} Oct 30 00:31:17.468510 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.468492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" event={"ID":"87b9cc45a336a6ed23d0ff1b116a4584","Type":"ContainerStarted","Data":"ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71"} Oct 30 00:31:17.484644 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.484628 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:31:17.485432 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.485417 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:31:17.485508 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.485442 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:31:17.485508 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.485452 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:31:17.547470 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:17.547448 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:17.647963 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:17.647901 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:31:17.713340 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.713324 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:31:17.727562 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.727545 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:31:17.747700 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.747657 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:31:17.752273 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.752253 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:31:17.753172 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.753155 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:31:17.774680 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:17.774656 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:31:18.306679 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.306651 2577 apiserver.go:52] "Watching apiserver" Oct 30 00:31:18.327689 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.327664 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:31:18.351195 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.351170 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:31:18.351563 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.351543 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h282r","openshift-dns/node-resolver-m75x5","openshift-multus/network-metrics-daemon-n4f84","openshift-ovn-kubernetes/ovnkube-node-8mdqj","openshift-image-registry/node-ca-5f7mc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal","openshift-multus/multus-qmmbh","kube-system/konnectivity-agent-5zk8x","kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm","openshift-cluster-node-tuning-operator/tuned-nwh2w","openshift-multus/multus-additional-cni-plugins-pbljz","openshift-network-diagnostics/network-check-target-f4q86","openshift-network-operator/iptables-alerter-cl8gl"] Oct 30 00:31:18.354296 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.354268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.356386 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.356364 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:18.356507 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.356442 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:18.358712 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.358617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.358712 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.358682 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:18.360944 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.360925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.365362 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.365338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.366567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.366530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.369358 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.369340 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.371238 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371194 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.371238 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371200 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.371580 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.371702 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.371636 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:18.371943 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\"" Oct 30 00:31:18.371943 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371928 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:31:18.372071 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.371965 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:31:18.373841 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.373825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.376209 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.376181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.378618 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.378600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.381375 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.381357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.382506 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.382381 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:31:18.382506 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.382442 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-78chp" Oct 30 00:31:18.428359 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.428337 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:31:18.441312 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.441417 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441417 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbbs\" (UniqueName: \"kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.441525 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.441525 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441525 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441632 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.441632 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.441632 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.441736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.441736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441844 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.441844 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.441844 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.442097 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442097 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.441942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442097 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442097 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442097 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442121 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhj5c\" (UniqueName: \"kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442240 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.442344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.442591 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442591 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442591 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjzq\" (UniqueName: \"kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442591 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnslc\" (UniqueName: \"kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.442591 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bdb\" (UniqueName: \"kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442631 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9rr\" (UniqueName: \"kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442729 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.442799 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.442977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.443289 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.443932 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444021 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.444021 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.443979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.444021 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.444158 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.444158 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444158 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444158 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tzgc\" (UniqueName: \"kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.444345 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.444345 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.444345 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444345 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.444532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqc5\" (UniqueName: \"kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.444809 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.445192 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.445192 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.444847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktzs\" (UniqueName: \"kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.459339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.459314 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:31:18.472217 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472196 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.472547 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472521 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.472642 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472580 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.472642 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472628 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.472732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472656 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.472732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472679 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\"" Oct 30 00:31:18.472732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472703 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:31:18.472732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472700 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472734 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472735 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472740 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472788 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472787 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472791 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472826 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472837 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472848 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472884 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472896 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472903 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472904 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472917 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472926 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:31:18.472928 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472946 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472961 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472976 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.472988 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.473034 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:31:18.473567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.473082 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\"" Oct 30 00:31:18.485829 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.485807 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:31:18.545603 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545603 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhj5c\" (UniqueName: \"kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545700 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545720 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545703 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545770 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.545797 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjzq\" (UniqueName: \"kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnslc\" (UniqueName: \"kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bdb\" (UniqueName: \"kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.545999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546022 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9rr\" (UniqueName: \"kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.546339 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546216 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546267 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546324 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546391 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547132 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546669 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzgc\" (UniqueName: \"kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.547975 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqc5\" (UniqueName: \"kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktzs\" (UniqueName: \"kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547418 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbbs\" (UniqueName: \"kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.548767 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.547971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.549600 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548061 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548088 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548422 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548450 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548554 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.546420 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.548684 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.048645853 +0000 UTC m=+3.061917528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.548963 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549043 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.550500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549098 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549140 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.549679 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.549730 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.049713001 +0000 UTC m=+3.062984668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549766 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549817 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549836 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.549872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.550112 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.550536 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.550635 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.551306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551036 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551275 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.546751 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551438 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551931 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.551998 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.552150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552087 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.552567 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.552782 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.552616 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.553253 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.553230 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.553803 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.553783 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.554731 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.554713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.571108 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.571057 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:18.571108 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.571078 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:18.571108 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.571090 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:18.571299 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:18.571146 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.071130755 +0000 UTC m=+3.084402411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:18.572722 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.572696 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktzs\" (UniqueName: \"kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.572962 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.572927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjzq\" (UniqueName: \"kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.574134 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.574094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhj5c\" (UniqueName: \"kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.576233 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.576210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbbs\" (UniqueName: \"kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.576313 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.576272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnslc\" (UniqueName: \"kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:18.577595 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.577574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzgc\" (UniqueName: \"kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.577674 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.577622 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9rr\" (UniqueName: \"kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.579612 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.579588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqc5\" (UniqueName: \"kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.589016 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.588998 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bdb\" (UniqueName: \"kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.672642 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.672622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmmbh" Oct 30 00:31:18.682330 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.682298 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79358397_9466_4f7b_9349_cd976b869ead.slice/crio-bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4 WatchSource:0}: Error finding container bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4: Status 404 returned error can't find the container with id bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4 Oct 30 00:31:18.695139 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.695111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:31:18.702150 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.702126 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:31:18.702218 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.702199 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:18.705639 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.705617 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56772ce8_a261_4951_b032_2b02e0f486ec.slice/crio-a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1 WatchSource:0}: Error finding container a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1: Status 404 returned error can't find the container with id a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1 Oct 30 00:31:18.705851 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.705777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:18.713479 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.713271 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ed2348_6522_439c_9381_92240fabb46d.slice/crio-5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b WatchSource:0}: Error finding container 5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b: Status 404 returned error can't find the container with id 5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b Oct 30 00:31:18.713479 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.713341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m75x5" Oct 30 00:31:18.721159 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.721083 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e4db94_de44_4a89_9bfe_5ac1d5997038.slice/crio-31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66 WatchSource:0}: Error finding container 31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66: Status 404 returned error can't find the container with id 31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66 Oct 30 00:31:18.721519 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.721493 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9caa4b_59b7_47c9_a7d5_0141bd0d25fc.slice/crio-2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61 WatchSource:0}: Error finding container 2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61: Status 404 returned error can't find the container with id 2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61 Oct 30 00:31:18.725228 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.725153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:31:18.727320 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.727293 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aaeeff_dda3_47f1_84cf_ea29bb1bd772.slice/crio-c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc WatchSource:0}: Error finding container c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc: Status 404 returned error can't find the container with id c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc Oct 30 00:31:18.728480 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.728461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:31:18.740782 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.740757 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf30350_e3f7_4edd_a691_b499f83bccb6.slice/crio-b0e737d06ccad445d86b435994329708227cd9717525abd56c78d45fef4c31e3 WatchSource:0}: Error finding container b0e737d06ccad445d86b435994329708227cd9717525abd56c78d45fef4c31e3: Status 404 returned error can't find the container with id b0e737d06ccad445d86b435994329708227cd9717525abd56c78d45fef4c31e3 Oct 30 00:31:18.750878 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:18.750841 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:31:18.761052 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:31:18.761023 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b8acd5_22da_46b5_a40d_7e45cafcc342.slice/crio-52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff WatchSource:0}: Error finding container 52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff: Status 404 returned error can't find the container with id 52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff Oct 30 00:31:19.050887 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.050829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:19.051045 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.050899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:19.051045 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.050994 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:19.051045 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.050998 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:19.051151 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.051048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:20.051032619 +0000 UTC m=+4.064304287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:19.051151 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.051062 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:20.051056233 +0000 UTC m=+4.064327886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:19.151572 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.151499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:19.151753 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.151735 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:19.151804 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.151760 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:19.151804 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.151774 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:19.151883 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.151833 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:20.151816368 +0000 UTC m=+4.165088025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:19.386249 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.386150 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:18 +0000 UTC" deadline="2028-04-08 07:44:15.281212542 +0000 UTC" Oct 30 00:31:19.386249 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.386206 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12631h12m55.895010627s" Oct 30 00:31:19.465048 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.464593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:19.465048 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:19.464703 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:19.490889 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.490727 2577 generic.go:356] "Generic (PLEG): container finished" podID="5b21b9005987db40e0d6504b950f5deb" containerID="137309faba7fea05098e6d9415726070ef5c0692681dead6b25c0fc1ddf2e974" exitCode=0 Oct 30 00:31:19.490889 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.490808 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerDied","Data":"137309faba7fea05098e6d9415726070ef5c0692681dead6b25c0fc1ddf2e974"} Oct 30 00:31:19.500134 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.499701 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" event={"ID":"87b9cc45a336a6ed23d0ff1b116a4584","Type":"ContainerStarted","Data":"f5383eb38ea54bfb44a337414777ba91ec54268919d4cf77259ca2256a034e47"} Oct 30 00:31:19.504303 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.504278 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m75x5" event={"ID":"34aaeeff-dda3-47f1-84cf-ea29bb1bd772","Type":"ContainerStarted","Data":"c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc"} Oct 30 00:31:19.514851 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.514826 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66"} Oct 30 00:31:19.537548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.537501 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl8gl" event={"ID":"73ed2348-6522-439c-9381-92240fabb46d","Type":"ContainerStarted","Data":"5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b"} Oct 30 00:31:19.576878 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.573038 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5f7mc" event={"ID":"56772ce8-a261-4951-b032-2b02e0f486ec","Type":"ContainerStarted","Data":"a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1"} Oct 30 00:31:19.594120 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.594090 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmmbh" event={"ID":"79358397-9466-4f7b-9349-cd976b869ead","Type":"ContainerStarted","Data":"bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4"} Oct 30 00:31:19.614066 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.614036 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerStarted","Data":"52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff"} Oct 30 00:31:19.622404 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.622379 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" event={"ID":"6bf30350-e3f7-4edd-a691-b499f83bccb6","Type":"ContainerStarted","Data":"b0e737d06ccad445d86b435994329708227cd9717525abd56c78d45fef4c31e3"} Oct 30 00:31:19.635967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.635943 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" event={"ID":"24461adc-8328-4dd9-a4ac-859666bdc68c","Type":"ContainerStarted","Data":"0562361d066cf1a82d021d9a41c9bfdbd4181c8717b7bef724ec3eecaea30a0e"} Oct 30 00:31:19.647753 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:19.647730 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5zk8x" event={"ID":"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc","Type":"ContainerStarted","Data":"2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61"} Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.056895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.056972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.057100 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.057157 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:22.057137733 +0000 UTC m=+6.070409404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.057548 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:20.057672 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.057595 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:22.057580526 +0000 UTC m=+6.070852194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:20.157652 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.157513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:20.157840 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.157693 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:20.157840 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.157721 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:20.157840 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.157733 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:20.157840 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.157787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:22.157769241 +0000 UTC m=+6.171040917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:20.386925 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.386812 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:18 +0000 UTC" deadline="2028-04-13 07:56:34.862801505 +0000 UTC" Oct 30 00:31:20.386925 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.386882 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12751h25m14.475923691s" Oct 30 00:31:20.465149 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.464656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:20.465149 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.464776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:20.465149 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.465139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:20.465412 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:20.465242 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:20.675406 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.675287 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerStarted","Data":"b7ca537036736941b52cbddc1be48522cded10595af2bb44f951cd491a4a6551"} Oct 30 00:31:20.715825 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:20.715763 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" podStartSLOduration=3.715745331 podStartE2EDuration="3.715745331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:19.570739376 +0000 UTC m=+3.584011053" watchObservedRunningTime="2026-10-30 00:31:20.715745331 +0000 UTC m=+4.729017008" Oct 30 00:31:21.465428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:21.464988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:21.465428 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:21.465114 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:22.071974 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.071885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:22.072116 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.072073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:22.072239 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.072220 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:22.072310 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.072295 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:26.072275032 +0000 UTC m=+10.085546689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:22.072702 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.072685 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:22.072786 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.072742 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:26.072725845 +0000 UTC m=+10.085997500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:22.173394 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.173356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:22.173564 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.173542 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:22.173564 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.173561 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:22.173667 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.173573 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:22.173667 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.173624 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:26.173607282 +0000 UTC m=+10.186878940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:22.464933 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.464685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:22.464933 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.464788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:22.465143 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.465121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:22.465271 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:22.465226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:22.690134 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.690102 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="ca738f142e06e179ed6c54b631ef96c25df8f9efa40dadd126e1033cef678447" exitCode=0 Oct 30 00:31:22.690488 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.690189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"ca738f142e06e179ed6c54b631ef96c25df8f9efa40dadd126e1033cef678447"} Oct 30 00:31:22.697053 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.697027 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5zk8x" event={"ID":"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc","Type":"ContainerStarted","Data":"4b99491003822e4cea86e7e7aa96bca4c77eabdc3c8dcba4d0f096e85037ae42"} Oct 30 00:31:22.767063 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.766972 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" podStartSLOduration=5.766957805 podStartE2EDuration="5.766957805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:20.716579153 +0000 UTC m=+4.729850833" watchObservedRunningTime="2026-10-30 00:31:22.766957805 +0000 UTC m=+6.780229481" Oct 30 00:31:22.807294 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:22.807235 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5zk8x" podStartSLOduration=3.049256782 podStartE2EDuration="6.807215829s" totalImagesPullingTime="3.757959047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:22.80665327 +0000 UTC m=+6.819924947" watchObservedRunningTime="2026-10-30 00:31:22.807215829 +0000 UTC m=+6.820487505" Oct 30 00:31:23.464914 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:23.464883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:23.465055 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:23.465013 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:23.702994 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:23.702919 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5f7mc" event={"ID":"56772ce8-a261-4951-b032-2b02e0f486ec","Type":"ContainerStarted","Data":"29f2895f5195bb270bf6c0fe40ef7e52eb04f32dad826040cc84e1423fbe950a"} Oct 30 00:31:23.741753 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:23.741700 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5f7mc" podStartSLOduration=3.070291417 podStartE2EDuration="7.74168144s" totalImagesPullingTime="4.671390023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:23.74120219 +0000 UTC m=+7.754473866" watchObservedRunningTime="2026-10-30 00:31:23.74168144 +0000 UTC m=+7.754953117" Oct 30 00:31:24.469650 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:24.468960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:24.469650 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:24.469084 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:24.469650 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:24.469416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:24.469650 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:24.469497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:24.707679 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:24.707609 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" event={"ID":"24461adc-8328-4dd9-a4ac-859666bdc68c","Type":"ContainerStarted","Data":"f4bf00dc303c79bd31c6c790229c984e06bf893fd03f3e3be8c43b2854df65af"} Oct 30 00:31:24.873395 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:24.872741 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:25.464321 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:25.464288 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:25.464503 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:25.464420 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:25.774441 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:25.774412 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.106512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.106570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.106707 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.106765 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.106748358 +0000 UTC m=+18.120020017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.107181 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:26.107269 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.107233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.107218036 +0000 UTC m=+18.120489693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:26.207813 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.207709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:26.207956 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.207831 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:26.207956 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.207850 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:26.207956 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.207887 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:26.207956 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.207955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:34.207938129 +0000 UTC m=+18.221209799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:26.393923 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.393729 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:31:25.774436516Z","UUID":"5e314361-4603-4a1a-bc1c-d7f012f5ab58","Handler":null,"Name":"","Endpoint":""} Oct 30 00:31:26.395412 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.395368 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:31:26.395412 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.395423 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:31:26.466003 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.464978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:26.466087 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.466056 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:26.466162 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.466148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:26.466249 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:26.466232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:26.714031 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.713950 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m75x5" event={"ID":"34aaeeff-dda3-47f1-84cf-ea29bb1bd772","Type":"ContainerStarted","Data":"5a6221489e34dded79bf1cfa6c6e3f10f3a212d5cedb2efa8460f1d977cd814d"} Oct 30 00:31:26.715839 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.715810 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmmbh" event={"ID":"79358397-9466-4f7b-9349-cd976b869ead","Type":"ContainerStarted","Data":"4a69f14ff50de88520816fdeff6a06820a3bbe7952d17374b0ddd77098733c2a"} Oct 30 00:31:26.718272 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.718245 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" event={"ID":"24461adc-8328-4dd9-a4ac-859666bdc68c","Type":"ContainerStarted","Data":"91c261d4ce9b7ef89f6c29cc195e12ba173f02ed08b7d1b4b116f6af91876deb"} Oct 30 00:31:26.753527 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.753466 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m75x5" podStartSLOduration=3.035845388 podStartE2EDuration="10.753450316s" totalImagesPullingTime="7.717604928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:26.753179333 +0000 UTC m=+10.766451012" watchObservedRunningTime="2026-10-30 00:31:26.753450316 +0000 UTC m=+10.766721994" Oct 30 00:31:26.795004 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.794743 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qmmbh" podStartSLOduration=3.365347086 podStartE2EDuration="10.794726231s" totalImagesPullingTime="7.429379145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:26.794172 +0000 UTC m=+10.807443688" watchObservedRunningTime="2026-10-30 00:31:26.794726231 +0000 UTC m=+10.807997907" Oct 30 00:31:26.920420 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.919619 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:26.920982 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:26.920806 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:27.464522 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.464448 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:27.464662 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:27.464569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:27.722485 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.722405 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cl8gl" event={"ID":"73ed2348-6522-439c-9381-92240fabb46d","Type":"ContainerStarted","Data":"7d24ef3029775a141cac9e4fea0bca8f64136671a55e9b4495afc50c62b1f91d"} Oct 30 00:31:27.724642 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.724614 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="a32c12d1ef59feffedef0d86da87674f447879d8de411e0378b73b267c6bb697" exitCode=0 Oct 30 00:31:27.724771 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.724694 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"a32c12d1ef59feffedef0d86da87674f447879d8de411e0378b73b267c6bb697"} Oct 30 00:31:27.729635 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.728749 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" event={"ID":"24461adc-8328-4dd9-a4ac-859666bdc68c","Type":"ContainerStarted","Data":"9ee9b281d615ee4e4e5eeffde504da3a5f3ee8c3a7cf89ac27a9c4b02280c233"} Oct 30 00:31:27.729972 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.729940 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:31:27.762214 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.761997 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cl8gl" podStartSLOduration=4.422583142 podStartE2EDuration="11.761984095s" totalImagesPullingTime="7.339400953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:27.761715207 +0000 UTC m=+11.774986883" watchObservedRunningTime="2026-10-30 00:31:27.761984095 +0000 UTC m=+11.775255771" Oct 30 00:31:27.841658 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:27.841608 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" podStartSLOduration=4.226363998 podStartE2EDuration="11.841593888s" totalImagesPullingTime="7.61522989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:27.841048175 +0000 UTC m=+11.854319852" watchObservedRunningTime="2026-10-30 00:31:27.841593888 +0000 UTC m=+11.854865563" Oct 30 00:31:28.466034 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:28.466001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:28.466202 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:28.466050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:28.466202 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:28.466161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:28.466582 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:28.466553 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:28.731411 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:28.731323 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" event={"ID":"6bf30350-e3f7-4edd-a691-b499f83bccb6","Type":"ContainerStarted","Data":"c5c5cf105c861a6c6332c3c5348c87b663a7404300bb7eaa512c41a1a9e9f5e0"} Oct 30 00:31:28.793212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:28.793171 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" podStartSLOduration=3.028981918 podStartE2EDuration="12.793157123s" totalImagesPullingTime="9.764175205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:28.7929562 +0000 UTC m=+12.806227875" watchObservedRunningTime="2026-10-30 00:31:28.793157123 +0000 UTC m=+12.806428799" Oct 30 00:31:29.464373 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:29.464339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:29.464955 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:29.464464 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:29.736252 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:29.736168 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="e5a47156cce88cf731a510f33c3e0418ad177b444dccf06842bee925ad97d617" exitCode=0 Oct 30 00:31:29.736400 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:29.736255 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"e5a47156cce88cf731a510f33c3e0418ad177b444dccf06842bee925ad97d617"} Oct 30 00:31:30.464436 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:30.464402 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:30.464949 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:30.464528 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:30.465195 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:30.465178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:30.465302 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:30.465284 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:31.464419 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:31.464388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:31.464589 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:31.464499 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:31.742735 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:31.742658 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="8fa833d8d05260462b1149ad2cfcd47116aa30ab87d960024f5fac091aafa7c0" exitCode=0 Oct 30 00:31:31.742735 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:31.742703 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"8fa833d8d05260462b1149ad2cfcd47116aa30ab87d960024f5fac091aafa7c0"} Oct 30 00:31:32.465368 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:32.465195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:32.465709 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:32.465219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:32.465709 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:32.465467 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:32.465709 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:32.465577 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:32.746896 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:32.746790 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"fe6620e495de7c37766c548d0cca9a2ce2b714c9bba799387b515c92fdb15a9e"} Oct 30 00:31:33.137413 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.137381 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:33.330484 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.330457 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:33.465171 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.465147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:33.465271 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:33.465233 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:33.750900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.750877 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:31:33.751321 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751266 2577 generic.go:356] "Generic (PLEG): container finished" podID="34e4db94-de44-4a89-9bfe-5ac1d5997038" containerID="f4dade5736965cb021327e2a9d58aa56910319b4af396024c9db38277c76edeb" exitCode=1 Oct 30 00:31:33.751362 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751336 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"ab4cf52b2629000c25963aa625e8972064911bff7356180b8d50f070d9517c88"} Oct 30 00:31:33.751408 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751393 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"6dbf502a65323d589dd1d8a4bb2e1aa71912f7196c043ed44bb0434eae699eec"} Oct 30 00:31:33.751444 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751434 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"0d41a62593f44153e655c7fd12def7420a0f78eea50b353362076af693ee570f"} Oct 30 00:31:33.751481 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751465 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"3bfa4494fb98168b376090c9d29a82df94a7ab6620bc7bd82ea8f317796dceb9"} Oct 30 00:31:33.751522 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.751492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerDied","Data":"f4dade5736965cb021327e2a9d58aa56910319b4af396024c9db38277c76edeb"} Oct 30 00:31:33.753976 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.753953 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="c205dbc6eecb34c0cae5a17334371ccfa539be4f8f64c268235f1896e6fd7c53" exitCode=0 Oct 30 00:31:33.754066 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.753991 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"c205dbc6eecb34c0cae5a17334371ccfa539be4f8f64c268235f1896e6fd7c53"} Oct 30 00:31:33.927907 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:33.927874 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:34.159167 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.159146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:34.159284 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.159180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:34.159284 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.159265 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:34.159284 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.159266 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:34.159405 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.159306 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:31:50.159295848 +0000 UTC m=+34.172567501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:34.159405 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.159320 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:50.159313259 +0000 UTC m=+34.172584913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:34.259402 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.259381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:34.259520 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.259509 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:34.259555 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.259524 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:34.259555 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.259532 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:34.259634 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.259568 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:31:50.25955805 +0000 UTC m=+34.272829704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:34.297676 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.297656 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:34.433645 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.433560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:34.464855 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.464833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:34.464969 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.464928 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:34.465037 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.465022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:34.465132 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:34.465114 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:34.759089 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.759021 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="e02b6dc23274bd31681f6ad62df106c2f10369e167774d07d9221d5d9ec6f9c6" exitCode=0 Oct 30 00:31:34.759089 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:34.759065 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerDied","Data":"e02b6dc23274bd31681f6ad62df106c2f10369e167774d07d9221d5d9ec6f9c6"} Oct 30 00:31:35.464555 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:35.464529 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:35.464679 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:35.464609 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:35.764142 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:35.764086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:31:35.764585 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:35.764464 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"d04b139cfe9eb63054abc7000b63dfbe06a68e2341f4845b721f773792fd58a4"} Oct 30 00:31:35.767729 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:35.767706 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbljz" event={"ID":"c2b8acd5-22da-46b5-a40d-7e45cafcc342","Type":"ContainerStarted","Data":"52a35ce6ef2ed1ecf8e3758432ebc0f5a5e2751c8e462372dbe76d926a3ee187"} Oct 30 00:31:35.830791 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:35.830750 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pbljz" podStartSLOduration=9.522721817 podStartE2EDuration="19.830738567s" totalImagesPullingTime="10.30801675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:35.830345917 +0000 UTC m=+19.843617592" watchObservedRunningTime="2026-10-30 00:31:35.830738567 +0000 UTC m=+19.844010243" Oct 30 00:31:36.467252 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:36.467228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:36.467393 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:36.467309 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:36.467393 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:36.467380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:36.467473 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:36.467457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:36.871344 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:36.871308 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:37.011803 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.011780 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:37.465142 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.465069 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:37.465290 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:37.465176 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:37.774518 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.774461 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:31:37.774798 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.774780 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"575a885d3363ac805fc6f5ebeacfde098bc66662f357b03d40cc5cebaf76b578"} Oct 30 00:31:37.775129 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.775102 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:37.775250 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.775116 2577 scope.go:122] "RemoveContainer" containerID="f4dade5736965cb021327e2a9d58aa56910319b4af396024c9db38277c76edeb" Oct 30 00:31:37.792336 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:37.792209 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:38.464238 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.464212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:38.464238 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.464232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:38.464912 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:38.464336 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:38.464912 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:38.464450 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:38.779822 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.779755 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:31:38.780167 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.780134 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" event={"ID":"34e4db94-de44-4a89-9bfe-5ac1d5997038","Type":"ContainerStarted","Data":"5c918e44e61b08d6ff2a6e6a82eb75928b31fbf8ca05980f64eda56d32ea718f"} Oct 30 00:31:38.780432 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.780416 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:38.780510 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.780504 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:38.795500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.795482 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:31:38.876522 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:38.876477 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" podStartSLOduration=9.055741821 podStartE2EDuration="22.876464827s" totalImagesPullingTime="13.820723006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:38.876417069 +0000 UTC m=+22.889688745" watchObservedRunningTime="2026-10-30 00:31:38.876464827 +0000 UTC m=+22.889736533" Oct 30 00:31:39.176178 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.176151 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h282r"] Oct 30 00:31:39.176394 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.176279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:39.176394 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:39.176353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:39.179270 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.179245 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4f84"] Oct 30 00:31:39.179392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.179378 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:39.179515 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:39.179497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:39.179727 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.179707 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4q86"] Oct 30 00:31:39.179803 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.179793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:39.179890 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:39.179874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:39.521689 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.521612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:39.664829 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.664795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:39.833348 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:39.833266 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:40.087126 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:40.087052 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:40.224329 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:40.224303 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:40.466848 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:40.466821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:40.466989 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:40.466941 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:40.467055 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:40.466995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:40.467094 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:40.467070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:41.062459 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:41.062426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:41.211249 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:41.211217 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:41.464579 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:41.464554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:41.464713 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:41.464641 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:41.610032 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:41.609996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:42.466956 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:42.466926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:42.467313 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:42.467059 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:42.467313 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:42.467100 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:42.467313 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:42.467183 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:43.464115 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:43.464090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m75x5_34aaeeff-dda3-47f1-84cf-ea29bb1bd772/dns-node-resolver/0.log" Oct 30 00:31:43.464337 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:43.464120 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:43.464337 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:43.464214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:44.063306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.063238 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5f7mc_56772ce8-a261-4951-b032-2b02e0f486ec/node-ca/0.log" Oct 30 00:31:44.420594 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.420549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:44.464838 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.464809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:44.465002 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:44.464954 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:44.465002 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.464991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:44.465116 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:44.465073 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:44.599716 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.599670 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:44.745264 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.745187 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:44.882382 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:44.882353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:45.464968 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:45.464942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:45.465338 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:45.465035 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:45.564253 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:45.564221 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:45.730362 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:45.730281 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:45.866609 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:45.866579 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:46.015753 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.015669 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:46.287883 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.287775 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:46.465771 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.465742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:46.466143 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:46.465850 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:46.466143 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.465983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:46.466143 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:46.466107 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:46.581158 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.581075 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:46.725094 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.725065 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:46.862651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:46.862622 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.009212 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.009190 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.154967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.154900 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.464561 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.464502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:47.464684 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:47.464586 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:47.484154 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.484131 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.628895 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.628829 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:47.798949 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:47.798890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:48.080962 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.080893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:48.279085 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.279057 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:48.464601 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.464573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:48.464725 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.464573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:48.464725 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:48.464680 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:48.464813 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:48.464729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:48.789627 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.789553 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:48.941892 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:48.941851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:49.292340 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:49.292312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:49.432266 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:49.432233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:49.465123 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:49.465104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:49.465207 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:49.465192 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:49.581014 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:49.580950 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:49.719148 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:49.719121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:50.071829 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.071802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:50.181327 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.181295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:50.181438 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.181346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:50.181478 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.181433 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:50.181478 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.181441 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:50.181538 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.181488 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret podName:9e691205-7020-4ddd-9fe6-61bb9d460d30 nodeName:}" failed. No retries permitted until 2026-10-30 00:32:22.18147451 +0000 UTC m=+66.194746180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret") pod "global-pull-secret-syncer-h282r" (UID: "9e691205-7020-4ddd-9fe6-61bb9d460d30") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:31:50.181538 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.181502 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs podName:661abd2b-095c-4b75-ae58-a5c2a82ad0ff nodeName:}" failed. No retries permitted until 2026-10-30 00:32:22.181495134 +0000 UTC m=+66.194766789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs") pod "network-metrics-daemon-n4f84" (UID: "661abd2b-095c-4b75-ae58-a5c2a82ad0ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:31:50.260083 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.260054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:50.282083 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.282052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:50.282203 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.282189 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:31:50.282244 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.282206 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:31:50.282244 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.282216 2577 projected.go:196] Error preparing data for projected volume kube-api-access-kp9tb for pod openshift-network-diagnostics/network-check-target-f4q86: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:50.282310 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.282258 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb podName:42079552-0ddd-4e83-85ef-604e435466eb nodeName:}" failed. No retries permitted until 2026-10-30 00:32:22.28224334 +0000 UTC m=+66.295515006 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kp9tb" (UniqueName: "kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb") pod "network-check-target-f4q86" (UID: "42079552-0ddd-4e83-85ef-604e435466eb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:31:50.464593 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.464574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:50.464682 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.464671 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:50.464777 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.464761 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:50.464895 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:50.464877 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:50.866331 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:50.866307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:51.088997 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:51.088970 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:51.465068 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:51.465044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:51.465196 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:51.465122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:51.468583 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:51.468560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:51.666542 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:51.666499 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:52.092306 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:52.092273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:52.265671 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:52.265643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:52.464361 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:52.464335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:52.464485 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:52.464448 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:52.464538 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:52.464505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:52.464610 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:52.464592 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:52.861589 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:52.861557 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.366748 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:53.366725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.464606 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:53.464584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:53.464723 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:53.464667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:53.502810 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:53.502791 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:53.665046 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:53.664990 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:54.060261 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:54.060197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:54.272528 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:54.272500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:54.464739 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:54.464713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:54.465165 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:54.464838 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:54.465165 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:54.464883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:54.465165 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:54.464958 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:54.860103 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:54.860074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:55.071895 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:55.071853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:55.255192 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:55.255132 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:55.464277 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:55.464252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:55.464396 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:55.464333 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:55.517433 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:55.517380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:55.873814 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:55.873784 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:56.018835 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:56.018809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:56.278383 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:56.278326 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:56.419349 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:56.419320 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:31:56.465293 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:56.465271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:56.465412 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:56.465341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:56.465470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:56.465443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:56.465560 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:56.465542 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:57.464294 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:57.464269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:57.464633 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:57.464350 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:31:58.464343 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:58.464310 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:31:58.464711 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:58.464447 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:31:58.464711 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:58.464482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:31:58.464711 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:58.464558 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:31:59.464729 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:31:59.464700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:31:59.465217 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:31:59.464792 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4q86" podUID="42079552-0ddd-4e83-85ef-604e435466eb" Oct 30 00:32:00.219518 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.219487 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:00.422120 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.422087 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:00.465148 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.465125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:00.465560 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.465128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:00.465560 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:32:00.465232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4f84" podUID="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" Oct 30 00:32:00.465560 ip-10-0-131-130 kubenswrapper[2577]: E1030 00:32:00.465337 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h282r" podUID="9e691205-7020-4ddd-9fe6-61bb9d460d30" Oct 30 00:32:00.564685 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.564645 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:00.747800 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.747775 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeReady" Oct 30 00:32:00.747921 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.747885 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:32:00.809009 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.808950 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-846tj"] Oct 30 00:32:00.832427 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.832406 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gptfr"] Oct 30 00:32:00.832618 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.832597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-846tj" Oct 30 00:32:00.848382 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.848363 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mcpcw"] Oct 30 00:32:00.848463 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.848442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:00.852327 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.850803 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:32:00.852327 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.851036 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\"" Oct 30 00:32:00.852327 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.851189 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:32:00.864500 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.864481 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-846tj"] Oct 30 00:32:00.864580 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.864517 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gptfr"] Oct 30 00:32:00.864580 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.864540 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mcpcw"] Oct 30 00:32:00.864643 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.864586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.949353 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949367 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949380 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949374 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949407 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949445 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:32:00.949461 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949450 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:32:00.949673 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949485 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:32:00.949673 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.949496 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:32:00.960957 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.960936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.961045 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.960962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.961045 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.960982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:00.961045 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.961045 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.961179 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbjc\" (UniqueName: \"kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:00.961228 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9pt\" (UniqueName: \"kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:00.961261 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:00.961315 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlsr\" (UniqueName: \"kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:00.961347 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:00.961402 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:00.961389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.062441 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.062441 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.062441 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbjc\" (UniqueName: \"kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.062614 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9pt\" (UniqueName: \"kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.062614 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.062614 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlsr\" (UniqueName: \"kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.062614 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.062614 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.062857 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.062857 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.062857 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.062693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.063044 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.063025 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.063133 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.063109 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.063236 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.063171 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.063236 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.063124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.063236 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.063205 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.066145 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.066123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.066145 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.066134 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.066266 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.066244 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.086851 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.086825 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbjc\" (UniqueName: \"kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.086992 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.086975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlsr\" (UniqueName: \"kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.087272 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.087256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9pt\" (UniqueName: \"kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.171596 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.171571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-846tj" Oct 30 00:32:01.193705 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.193685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:01.248211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.247997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:01.465060 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.465037 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:01.481757 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.481739 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\"" Oct 30 00:32:01.482141 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.482123 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:32:01.482204 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.482191 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:32:01.549894 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.549853 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gptfr"] Oct 30 00:32:01.551153 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:01.551133 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7003a5d_b3e4_4c2e_88d2_f362a5f78180.slice/crio-1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642 WatchSource:0}: Error finding container 1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642: Status 404 returned error can't find the container with id 1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642 Oct 30 00:32:01.562170 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.562153 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-846tj"] Oct 30 00:32:01.563098 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:01.563078 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08604272_a8a8_44c9_bcc1_9b3f12a37d77.slice/crio-a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4 WatchSource:0}: Error finding container a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4: Status 404 returned error can't find the container with id a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4 Oct 30 00:32:01.582122 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.582105 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mcpcw"] Oct 30 00:32:01.583753 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:01.583714 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79858a62_0cea_444e_a2ce_fec46d542e55.slice/crio-7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59 WatchSource:0}: Error finding container 7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59: Status 404 returned error can't find the container with id 7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59 Oct 30 00:32:01.825610 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.825560 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gptfr" event={"ID":"c7003a5d-b3e4-4c2e-88d2-f362a5f78180","Type":"ContainerStarted","Data":"1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642"} Oct 30 00:32:01.826964 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.826942 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcpcw" event={"ID":"79858a62-0cea-444e-a2ce-fec46d542e55","Type":"ContainerStarted","Data":"b1d9dd461c45b77c5d8d9d79d48f8453a429510b57d493de32a70af0b9a99293"} Oct 30 00:32:01.827046 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.826992 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcpcw" event={"ID":"79858a62-0cea-444e-a2ce-fec46d542e55","Type":"ContainerStarted","Data":"7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59"} Oct 30 00:32:01.830544 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:01.830524 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-846tj" event={"ID":"08604272-a8a8-44c9-bcc1-9b3f12a37d77","Type":"ContainerStarted","Data":"a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4"} Oct 30 00:32:02.464738 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.464701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:02.465918 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.465456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:02.499517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.499338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:32:02.499517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.499407 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\"" Oct 30 00:32:02.499517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.499424 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\"" Oct 30 00:32:02.499517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.499496 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:32:02.834810 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:02.834735 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcpcw" event={"ID":"79858a62-0cea-444e-a2ce-fec46d542e55","Type":"ContainerStarted","Data":"f58fa2e8642103290ab4f694ef0b23481324d0e41ccd60014a8ea8db2675009c"} Oct 30 00:32:03.411277 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.411181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.492230 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.492204 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl"] Oct 30 00:32:03.494736 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.494712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.525153 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.525129 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 00:32:03.525153 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.525129 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:32:03.525504 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.525142 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:32:03.525504 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.525287 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 00:32:03.525989 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.525969 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl"] Oct 30 00:32:03.528489 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.528469 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8pprt\"" Oct 30 00:32:03.538878 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.538827 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:32:03.551846 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.551708 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.584211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.583157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.584211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.583219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.584211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.583265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.584211 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.583311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfd7\" (UniqueName: \"kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.683796 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.683715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.683796 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.683755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.683796 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.683785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.684041 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.683812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfd7\" (UniqueName: \"kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.684695 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.684672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.686702 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.686677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.687463 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.687440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.699480 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.699407 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:03.708024 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.708000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfd7\" (UniqueName: \"kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.808974 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.808947 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:03.839948 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.839922 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-846tj" event={"ID":"08604272-a8a8-44c9-bcc1-9b3f12a37d77","Type":"ContainerStarted","Data":"af7d64d9cde5fae8e05e68c0d7fb7b6c3fdc8eb19b260539ef9321cdbd5eef3e"} Oct 30 00:32:03.840064 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.839976 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-846tj" event={"ID":"08604272-a8a8-44c9-bcc1-9b3f12a37d77","Type":"ContainerStarted","Data":"f3277e4a5746e213db182ad1f11485896ee2669b530121986f4a125b70c5581f"} Oct 30 00:32:03.840542 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.840522 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-846tj" Oct 30 00:32:03.887182 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:03.887130 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-846tj" podStartSLOduration=1.998519342 podStartE2EDuration="3.88711316s" totalImagesPullingTime="1.888593818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:03.884826497 +0000 UTC m=+47.898098171" watchObservedRunningTime="2026-10-30 00:32:03.88711316 +0000 UTC m=+47.900384837" Oct 30 00:32:04.037340 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.037312 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:04.132980 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:04.132762 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9149ff3_efaf_4600_b5fe_7699ffcd8db5.slice/crio-7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638 WatchSource:0}: Error finding container 7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638: Status 404 returned error can't find the container with id 7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638 Oct 30 00:32:04.133732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.133694 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl"] Oct 30 00:32:04.170086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.170054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:04.306043 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.306019 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:04.453936 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.453899 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:04.844581 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.844551 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gptfr" event={"ID":"c7003a5d-b3e4-4c2e-88d2-f362a5f78180","Type":"ContainerStarted","Data":"0111ea71cf4d931619c818a6c2b39f7bef0d823111cfc2a682f255baf6aa005b"} Oct 30 00:32:04.846927 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.846896 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcpcw" event={"ID":"79858a62-0cea-444e-a2ce-fec46d542e55","Type":"ContainerStarted","Data":"9e80f4dafc600932b07b4388f19534c54ee5e0cd126f214aa49fd5869e7f9dd7"} Oct 30 00:32:04.848084 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.848057 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" event={"ID":"c9149ff3-efaf-4600-b5fe-7699ffcd8db5","Type":"ContainerStarted","Data":"7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638"} Oct 30 00:32:04.887497 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.887451 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gptfr" podStartSLOduration=2.158812032 podStartE2EDuration="4.88743848s" totalImagesPullingTime="2.728626448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:04.886551517 +0000 UTC m=+48.899823194" watchObservedRunningTime="2026-10-30 00:32:04.88743848 +0000 UTC m=+48.900710156" Oct 30 00:32:04.933170 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:04.933126 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mcpcw" podStartSLOduration=2.703000028 podStartE2EDuration="4.933112389s" totalImagesPullingTime="2.230112361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:32:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:04.931853694 +0000 UTC m=+48.945125372" watchObservedRunningTime="2026-10-30 00:32:04.933112389 +0000 UTC m=+48.946384062" Oct 30 00:32:05.853933 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:05.853891 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" event={"ID":"c9149ff3-efaf-4600-b5fe-7699ffcd8db5","Type":"ContainerStarted","Data":"a999b9d69bc81aaa5ddaeb138ec92d0f13f7f56006ab8ff3cac6ea852bece4d1"} Oct 30 00:32:05.854347 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:05.853989 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" event={"ID":"c9149ff3-efaf-4600-b5fe-7699ffcd8db5","Type":"ContainerStarted","Data":"cf0df8077d0694f9c39e65969ea99537d67ccb8fda3f0027d2cb0f8c6ac44ac4"} Oct 30 00:32:05.899034 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:05.898990 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" podStartSLOduration=1.7328518609999999 podStartE2EDuration="2.898976857s" totalImagesPullingTime="1.166124996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:05.898071299 +0000 UTC m=+49.911342974" watchObservedRunningTime="2026-10-30 00:32:05.898976857 +0000 UTC m=+49.912248532" Oct 30 00:32:06.079565 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:06.079532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:07.005834 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:07.005794 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:07.174046 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:07.173998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:07.307944 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:07.307846 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:07.876409 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:07.876373 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.035139 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.035107 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.110987 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.110958 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6"] Oct 30 00:32:08.113234 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.113214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.134555 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.134499 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6"] Oct 30 00:32:08.135701 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.135666 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 00:32:08.135811 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.135730 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:32:08.146804 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.146782 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-skp7j\"" Oct 30 00:32:08.167734 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.167708 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q"] Oct 30 00:32:08.169708 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.169688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.182300 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.182282 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzdmx"] Oct 30 00:32:08.184218 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.184203 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.192878 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.192842 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q"] Oct 30 00:32:08.198777 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.198760 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:32:08.199008 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.198988 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qncjw\"" Oct 30 00:32:08.199072 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.198997 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 00:32:08.214004 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.213978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.214103 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.214017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.214103 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.214057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4859\" (UniqueName: \"kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.214204 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.214181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.214835 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.214815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.235014 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.234994 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:32:08.235255 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.235241 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:32:08.235371 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.235357 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w25sk\"" Oct 30 00:32:08.238266 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.238250 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:32:08.238820 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.238807 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 00:32:08.314930 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.314910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4859\" (UniqueName: \"kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.315036 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.314939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315036 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.314962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315036 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.314980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315036 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.314998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.315240 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315290 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2gsf\" (UniqueName: \"kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315359 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315415 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkgx\" (UniqueName: \"kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315415 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315415 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.315532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.315532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315532 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315701 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315764 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.315764 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315891 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.315891 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.315858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.316174 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.316152 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.317361 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.317341 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.317585 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.317569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.339208 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.339192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4859\" (UniqueName: \"kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.363316 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.363293 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.417055 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gsf\" (UniqueName: \"kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417055 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkgx\" (UniqueName: \"kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417160 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417186 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417470 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417517 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.417900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417724 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.417900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.418116 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.418116 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.417996 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.418222 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.418202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.418334 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.418315 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.419649 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.419624 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.419747 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.419733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.419894 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.419875 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.419934 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.419893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.425694 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.425676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:08.440856 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.440829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkgx\" (UniqueName: \"kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.441288 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.441268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gsf\" (UniqueName: \"kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.481246 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.481216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:08.495804 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.495781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:08.504598 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.504566 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.506788 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:08.506759 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6283ace2_25a6_43fe_b12c_1d952c5c95da.slice/crio-91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f WatchSource:0}: Error finding container 91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f: Status 404 returned error can't find the container with id 91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f Oct 30 00:32:08.644392 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.644362 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.728011 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.727988 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6"] Oct 30 00:32:08.729246 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:08.729223 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d34e3a_70d1_43c0_9a10_8f8be0b4dbfc.slice/crio-04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6 WatchSource:0}: Error finding container 04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6: Status 404 returned error can't find the container with id 04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6 Oct 30 00:32:08.777987 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.777964 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q"] Oct 30 00:32:08.779497 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:08.779474 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ed79ad_9a13_4899_8307_eff21d88d3e6.slice/crio-dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232 WatchSource:0}: Error finding container dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232: Status 404 returned error can't find the container with id dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232 Oct 30 00:32:08.865135 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.865108 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" event={"ID":"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc","Type":"ContainerStarted","Data":"7c8406183bf601fa27217627a2233ea242a3e41132710eb082176e58ac831456"} Oct 30 00:32:08.865261 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.865171 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" event={"ID":"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc","Type":"ContainerStarted","Data":"fdfe8921c769100fe3060a9537fb71108750d00c9986f33f37f14f05ac143e8c"} Oct 30 00:32:08.865261 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.865204 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" event={"ID":"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc","Type":"ContainerStarted","Data":"04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6"} Oct 30 00:32:08.866233 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.866209 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzdmx" event={"ID":"6283ace2-25a6-43fe-b12c-1d952c5c95da","Type":"ContainerStarted","Data":"91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f"} Oct 30 00:32:08.867200 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.867174 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" event={"ID":"d4ed79ad-9a13-4899-8307-eff21d88d3e6","Type":"ContainerStarted","Data":"dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232"} Oct 30 00:32:08.882831 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.882795 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:08.956805 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.956783 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:08.958607 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.958591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:08.975734 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.975706 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:32:08.975905 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.975858 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:32:08.976010 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.975932 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q4g\"" Oct 30 00:32:08.976010 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.975993 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:32:08.976185 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.976171 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:32:08.976243 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.976232 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:32:08.990564 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.990506 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:32:08.990564 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.990546 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:32:08.990706 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.990658 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:32:08.992044 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:08.992027 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:09.021827 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.021808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.021941 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.021836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.021941 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.021874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.022016 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.021990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.022016 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.022010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.022100 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.022062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.022161 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.022148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.022205 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.022175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123287 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123263 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.123663 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.124026 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.123733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.124583 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.124553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.124682 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.124553 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.124744 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.124685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.124744 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.124699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.126338 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.126317 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.126495 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.126463 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.148283 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.148264 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.270409 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.270329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:09.360472 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.360436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:09.526602 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.526539 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:09.587738 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.587702 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:09.592145 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.592122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.598307 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.597328 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:09.602886 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:09.602840 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54406dd0_d002_4fc9_84b4_41532b139761.slice/crio-45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8 WatchSource:0}: Error finding container 45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8: Status 404 returned error can't find the container with id 45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8 Oct 30 00:32:09.613683 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.613665 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5vxhs\"" Oct 30 00:32:09.613903 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.613848 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:32:09.614084 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.614057 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:32:09.614167 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.614144 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:32:09.618165 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.616974 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:32:09.618690 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.618668 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:32:09.619636 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.619617 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:09.683223 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.683070 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:09.709074 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.709043 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:32:09.709203 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.709164 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:32:09.709269 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.709241 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:32:09.709269 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.709241 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:32:09.729727 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.729841 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.729841 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.729841 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.729992 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730111 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.729999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730111 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730111 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730111 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730363 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730363 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730363 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.730363 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.730205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831203 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831308 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831308 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831308 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831449 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831449 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831449 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831593 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831593 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831593 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831593 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831812 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.831812 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.831642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.832378 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.832349 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.832690 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.832594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.833116 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.833071 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.835554 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.835529 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.835827 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.835801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.835961 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.835943 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.836430 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.836398 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.836505 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.836478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.837713 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.837691 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.837806 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.837736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.837974 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.837899 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.838038 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.837993 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.856883 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.856843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:09.875000 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.874974 2577 generic.go:356] "Generic (PLEG): container finished" podID="6283ace2-25a6-43fe-b12c-1d952c5c95da" containerID="38b9766e6dd7fc88e7eb9e8c485a2b3412824e4c2808ae40f128cd420d90d0c4" exitCode=0 Oct 30 00:32:09.875095 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.875051 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzdmx" event={"ID":"6283ace2-25a6-43fe-b12c-1d952c5c95da","Type":"ContainerDied","Data":"38b9766e6dd7fc88e7eb9e8c485a2b3412824e4c2808ae40f128cd420d90d0c4"} Oct 30 00:32:09.876440 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.876411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc9dd444-ng59m" event={"ID":"54406dd0-d002-4fc9-84b4-41532b139761","Type":"ContainerStarted","Data":"45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8"} Oct 30 00:32:09.932823 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:09.932795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:10.087228 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.087194 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:10.246379 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.246346 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:10.277770 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.277747 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:10.290846 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:10.290822 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7063ed5c_d12e_4887_812f_4e1eeea42495.slice/crio-bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f WatchSource:0}: Error finding container bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f: Status 404 returned error can't find the container with id bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f Oct 30 00:32:10.797503 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.796400 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:10.883331 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.883297 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzdmx" event={"ID":"6283ace2-25a6-43fe-b12c-1d952c5c95da","Type":"ContainerStarted","Data":"cc322f873879d42e4ee1cf149b82e219f6541f4e75952c27090fd907c1f6de5b"} Oct 30 00:32:10.883486 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.883356 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzdmx" event={"ID":"6283ace2-25a6-43fe-b12c-1d952c5c95da","Type":"ContainerStarted","Data":"3922e5b470cffc23c030489e7ab3e5438ccad9e68e9326684d7dbfb062a1d411"} Oct 30 00:32:10.886485 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.886457 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" event={"ID":"d4ed79ad-9a13-4899-8307-eff21d88d3e6","Type":"ContainerStarted","Data":"95e9023494dd48f0e1dccc23540c53b8c2abb9441397743ce25716a4c0c2ce24"} Oct 30 00:32:10.886606 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.886513 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" event={"ID":"d4ed79ad-9a13-4899-8307-eff21d88d3e6","Type":"ContainerStarted","Data":"c97007f568f965ea30b944d8478fa9957873a6682640e9883a6e7a18c026268f"} Oct 30 00:32:10.886606 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.886542 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" event={"ID":"d4ed79ad-9a13-4899-8307-eff21d88d3e6","Type":"ContainerStarted","Data":"b653aedea67fdd16e952446cf1680a6fc889327e45e92b02e4d5a97e08d62efa"} Oct 30 00:32:10.888521 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.888494 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f"} Oct 30 00:32:10.890987 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.890952 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" event={"ID":"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc","Type":"ContainerStarted","Data":"bea8aa1b163fb1ef8c5d4bf8f9e9769787780ea4a69fe5890832f9a0e2a8072c"} Oct 30 00:32:10.933528 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.933482 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mzdmx" podStartSLOduration=1.864001879 podStartE2EDuration="2.933467209s" totalImagesPullingTime="986.479751ms" totalInitContainerRuntime="82.985579ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:10.931912081 +0000 UTC m=+54.945183759" watchObservedRunningTime="2026-10-30 00:32:10.933467209 +0000 UTC m=+54.946738887" Oct 30 00:32:10.982357 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:10.980835 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" podStartSLOduration=1.681605517 podStartE2EDuration="2.980819966s" totalImagesPullingTime="1.299214449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:10.980715025 +0000 UTC m=+54.993986705" watchObservedRunningTime="2026-10-30 00:32:10.980819966 +0000 UTC m=+54.994091639" Oct 30 00:32:11.027517 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.027464 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" podStartSLOduration=2.034200903 podStartE2EDuration="3.027446381s" totalImagesPullingTime="993.245478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:11.025834724 +0000 UTC m=+55.039106400" watchObservedRunningTime="2026-10-30 00:32:11.027446381 +0000 UTC m=+55.040718059" Oct 30 00:32:11.446665 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.446624 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:11.582958 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.582855 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:11.815241 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.815199 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:11.896172 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.896143 2577 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039" exitCode=0 Oct 30 00:32:11.896335 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.896272 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039"} Oct 30 00:32:11.954188 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:11.954146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:12.716900 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.716847 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56f7b49654-f62f5"] Oct 30 00:32:12.719938 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.719914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.736677 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.736657 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 00:32:12.736791 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.736758 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 00:32:12.749886 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.749810 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-78xdz\"" Oct 30 00:32:12.749988 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.749917 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1huu4egidotfk\"" Oct 30 00:32:12.749988 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.749939 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:32:12.749988 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.749943 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 00:32:12.751895 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.751858 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56f7b49654-f62f5"] Oct 30 00:32:12.860889 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.860856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861024 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.860922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861091 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.861026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5jrm\" (UniqueName: \"kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861091 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.861071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861188 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.861107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861188 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.861162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.861284 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.861190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.903751 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.903680 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc9dd444-ng59m" event={"ID":"54406dd0-d002-4fc9-84b4-41532b139761","Type":"ContainerStarted","Data":"568215ff2a76e0313bd49e6a1bafb330a77e7ce1f2b160d5912b9682433abeb1"} Oct 30 00:32:12.949952 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.949895 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6ccc9dd444-ng59m" podStartSLOduration=2.271893172 podStartE2EDuration="4.94985079s" totalImagesPullingTime="2.677957618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:12.948978439 +0000 UTC m=+56.962250112" watchObservedRunningTime="2026-10-30 00:32:12.94985079 +0000 UTC m=+56.963122466" Oct 30 00:32:12.962515 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962647 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jrm\" (UniqueName: \"kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962647 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962647 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962833 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962833 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.962833 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.962792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.963119 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.963072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.963411 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.963387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.963742 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.963718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.965086 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.965063 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.965486 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.965464 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.965518 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.965471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.982395 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.982348 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v"] Oct 30 00:32:12.985193 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.985177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:12.989463 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.989446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jrm\" (UniqueName: \"kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:12.990617 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:12.990595 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:13.003381 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.003360 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 00:32:13.003528 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.003394 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v"] Oct 30 00:32:13.003589 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.003539 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x662k\"" Oct 30 00:32:13.031843 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.031819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:13.065166 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.064559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw2w\" (UniqueName: \"kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.065166 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.064632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.122299 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.122267 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:13.166368 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.166052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw2w\" (UniqueName: \"kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.166368 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.166122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.168800 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.168770 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.190737 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.190718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw2w\" (UniqueName: \"kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.303415 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.303339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:13.326454 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.326374 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:13.348428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.348401 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56f7b49654-f62f5"] Oct 30 00:32:13.617893 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.614332 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v"] Oct 30 00:32:13.620314 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:13.620279 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7876ee6c_178d_4ca6_923a_d76c0867bd85.slice/crio-f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d WatchSource:0}: Error finding container f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d: Status 404 returned error can't find the container with id f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d Oct 30 00:32:13.909213 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.909183 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" event={"ID":"7876ee6c-178d-4ca6-923a-d76c0867bd85","Type":"ContainerStarted","Data":"f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d"} Oct 30 00:32:13.910519 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.910492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" event={"ID":"3757f105-2260-48d2-be57-41ba8481ae63","Type":"ContainerStarted","Data":"46fe183b7123e4c66e570906720c9e0266ff2bef4b8f505f9b59dd5cddcf5bc2"} Oct 30 00:32:13.914189 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.914160 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c"} Oct 30 00:32:13.914300 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.914232 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1"} Oct 30 00:32:13.914300 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.914276 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d"} Oct 30 00:32:13.914409 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.914317 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e"} Oct 30 00:32:13.914409 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:13.914344 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905"} Oct 30 00:32:14.856854 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:14.856815 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-846tj" Oct 30 00:32:14.922640 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:14.922607 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerStarted","Data":"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b"} Oct 30 00:32:14.925771 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:14.925738 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" event={"ID":"3757f105-2260-48d2-be57-41ba8481ae63","Type":"ContainerStarted","Data":"66d30d9958f9114f4457963ecab52de8430beaf29193ef387d1411f80f3ce9fb"} Oct 30 00:32:14.983458 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:14.983417 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.718907218 podStartE2EDuration="5.983403662s" totalImagesPullingTime="3.249031173s" totalInitContainerRuntime="15.465271ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:14.980079971 +0000 UTC m=+58.993351648" watchObservedRunningTime="2026-10-30 00:32:14.983403662 +0000 UTC m=+58.996675337" Oct 30 00:32:15.028879 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.028825 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" podStartSLOduration=1.643085259 podStartE2EDuration="3.028811276s" totalImagesPullingTime="1.385726017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:15.028046278 +0000 UTC m=+59.041317955" watchObservedRunningTime="2026-10-30 00:32:15.028811276 +0000 UTC m=+59.042082952" Oct 30 00:32:15.100571 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.100535 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:15.103513 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.103483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.115061 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.114836 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:15.120483 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.120449 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:32:15.120605 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.120527 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:32:15.120605 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.120589 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:32:15.136473 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136216 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:32:15.136473 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136376 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:32:15.136473 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136453 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:32:15.136684 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136544 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:32:15.136684 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136603 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:32:15.136684 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136257 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:32:15.136791 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136760 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:32:15.137009 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.136989 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:32:15.151651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.148112 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kz97h\"" Oct 30 00:32:15.151651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.148132 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:32:15.151651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.148173 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:32:15.151651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.148184 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:32:15.151651 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.148496 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ju4vrtprvtbp\"" Oct 30 00:32:15.155175 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.155151 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:15.190310 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190417 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190417 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190509 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190509 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190570 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190570 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190638 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190638 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190698 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190730 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190759 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190759 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190839 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190839 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190959 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.190959 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.191053 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.190973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.191053 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.191002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.191053 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.191026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291408 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291408 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291582 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.291967 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.291668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292680 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292680 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292680 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292915 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292915 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292915 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292915 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.292915 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.292804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.294930 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.293241 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.294930 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.293516 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.294930 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.294051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.294930 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.294601 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.295697 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.295178 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.295697 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.295241 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.295697 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.295374 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.295935 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.295778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.295935 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.295895 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.296350 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.296324 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.296828 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.296800 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.296947 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.296832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.297052 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.297022 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.297146 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.297082 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.297224 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.297202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.297768 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.297748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.298170 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.298154 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.314583 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.314543 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:15.317357 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.317334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.427617 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.427589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:15.767981 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.767895 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:15.930437 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.930409 2577 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="72e98b4e42c17edf9f57b20b784ea8c84a222392e3cad7636cc6ee0b10458141" exitCode=0 Oct 30 00:32:15.930786 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.930498 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"72e98b4e42c17edf9f57b20b784ea8c84a222392e3cad7636cc6ee0b10458141"} Oct 30 00:32:15.930786 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.930583 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"e35688a19a204bedae8fdeab3162e4e319d7fe5b161ee7e99aebd7d81c8f801f"} Oct 30 00:32:15.932210 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.932185 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" event={"ID":"7876ee6c-178d-4ca6-923a-d76c0867bd85","Type":"ContainerStarted","Data":"62602e62095ff6264d1db360f9f68a4f4ed68b0faa002a965219775bb6936e89"} Oct 30 00:32:15.932531 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.932494 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:15.937120 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:15.937100 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:16.035905 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:16.035825 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" podStartSLOduration=2.706634125 podStartE2EDuration="4.03581371s" totalImagesPullingTime="1.329179585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:16.035334766 +0000 UTC m=+60.048606444" watchObservedRunningTime="2026-10-30 00:32:16.03581371 +0000 UTC m=+60.049085383" Oct 30 00:32:16.605907 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:16.605873 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:16.606258 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:16.606234 2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:16.786000 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:16.785962 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:19.271021 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:19.270993 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:19.334109 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:19.334083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:19.555504 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:19.555433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:19.949362 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:19.949329 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"880d9d2fda5b9d1a6f1ebb9327ab0c092aa91827439a24ee70bc95569e9ea30a"} Oct 30 00:32:19.949522 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:19.949415 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"b8d357332c37156f88eec1234009fb5c9d943d313fd513c2b1093982bff96e90"} Oct 30 00:32:20.098290 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.098255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:20.100658 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.100630 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:32:20.103512 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.103482 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.132974 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.132949 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:32:20.139732 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.139707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.139887 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.139746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.139887 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.139772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.140003 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.139972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.140067 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.140050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.140113 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.140088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.140147 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.140119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.140196 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.140182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.241677 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.241677 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.241677 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.241904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.242548 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.242524 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.244148 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.243254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.244148 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.243420 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.244148 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.244106 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.246495 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.246473 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.247078 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.247058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.249583 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.249561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.273084 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.273060 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.356837 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.356806 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:20.418166 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.418140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:20.727493 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.727453 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:32:20.728388 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:20.728362 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fb59b0_3c5a_4a80_9ddc_c394e1dcd8fc.slice/crio-d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9 WatchSource:0}: Error finding container d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9: Status 404 returned error can't find the container with id d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9 Oct 30 00:32:20.959423 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.959386 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546cc56848-dbr26" event={"ID":"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc","Type":"ContainerStarted","Data":"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d"} Oct 30 00:32:20.959615 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:20.959448 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546cc56848-dbr26" event={"ID":"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc","Type":"ContainerStarted","Data":"d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9"} Oct 30 00:32:21.603185 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:21.603152 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:21.966713 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:21.966639 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"8cc5d49e2b1d6cb95d6f24a6335f407c5a76e1490cf548eab0e930614c699b2e"} Oct 30 00:32:21.966713 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:21.966712 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"125daffba8f0319d9001ff92d63841ff6f772d9519d761d6d1702713e1155818"} Oct 30 00:32:21.966953 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:21.966738 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"91d348f596e61e2651462a869681353be154d365933d0e2c8762c2aeaaa5f42b"} Oct 30 00:32:21.966953 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:21.966764 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerStarted","Data":"bccaedccc1d4dda885b59f87d1aa89987ee160780cadd0e0419558d2fbcef8a1"} Oct 30 00:32:22.076731 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.076687 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.808198005 podStartE2EDuration="7.076673688s" totalImagesPullingTime="5.252031274s" totalInitContainerRuntime="16.444409ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:22.062687276 +0000 UTC m=+66.075958965" watchObservedRunningTime="2026-10-30 00:32:22.076673688 +0000 UTC m=+66.089945363" Oct 30 00:32:22.077733 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.077700 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-546cc56848-dbr26" podStartSLOduration=2.077687078 podStartE2EDuration="2.077687078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:21.008880879 +0000 UTC m=+65.022152556" watchObservedRunningTime="2026-10-30 00:32:22.077687078 +0000 UTC m=+66.090958755" Oct 30 00:32:22.262201 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.262121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:22.262333 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.262243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:22.277729 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.277706 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:32:22.278542 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.278525 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:32:22.285085 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.285054 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:22.285197 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.285105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:22.302099 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.302080 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\"" Oct 30 00:32:22.307162 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.307146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:22.363966 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.363464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:22.378066 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.378025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:22.379000 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.378985 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:32:22.399195 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.399174 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:32:22.406494 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.406472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:22.509676 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.509647 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\"" Oct 30 00:32:22.514822 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.514771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:22.528885 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.528842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:22.595800 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.595754 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\"" Oct 30 00:32:22.601228 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.600763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:22.609642 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:22.609618 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661abd2b_095c_4b75_ae58_a5c2a82ad0ff.slice/crio-6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88 WatchSource:0}: Error finding container 6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88: Status 404 returned error can't find the container with id 6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88 Oct 30 00:32:22.609938 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.609920 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4f84"] Oct 30 00:32:22.818351 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.818315 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4q86"] Oct 30 00:32:22.819658 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:22.819629 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42079552_0ddd_4e83_85ef_604e435466eb.slice/crio-dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48 WatchSource:0}: Error finding container dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48: Status 404 returned error can't find the container with id dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48 Oct 30 00:32:22.901892 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.900463 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h282r"] Oct 30 00:32:22.904447 ip-10-0-131-130 kubenswrapper[2577]: W1030 00:32:22.904424 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e691205_7020_4ddd_9fe6_61bb9d460d30.slice/crio-8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6 WatchSource:0}: Error finding container 8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6: Status 404 returned error can't find the container with id 8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6 Oct 30 00:32:22.971671 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.971642 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h282r" event={"ID":"9e691205-7020-4ddd-9fe6-61bb9d460d30","Type":"ContainerStarted","Data":"8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6"} Oct 30 00:32:22.973658 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.973631 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4q86" event={"ID":"42079552-0ddd-4e83-85ef-604e435466eb","Type":"ContainerStarted","Data":"dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48"} Oct 30 00:32:22.975345 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:22.975304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4f84" event={"ID":"661abd2b-095c-4b75-ae58-a5c2a82ad0ff","Type":"ContainerStarted","Data":"6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88"} Oct 30 00:32:23.240143 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:23.240110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:23.381588 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:23.381542 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:23.981450 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:23.981411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4f84" event={"ID":"661abd2b-095c-4b75-ae58-a5c2a82ad0ff","Type":"ContainerStarted","Data":"82ee33c2e9210c3abfc7639a5e4d024fbae4a165fd3f95961d20ba523a0fefc7"} Oct 30 00:32:23.981916 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:23.981471 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4f84" event={"ID":"661abd2b-095c-4b75-ae58-a5c2a82ad0ff","Type":"ContainerStarted","Data":"4a1a21e43f2bea91f0c7a8522f47c3227a837a3c363f45b15ca3ce58c522fd12"} Oct 30 00:32:24.036538 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:24.036481 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n4f84" podStartSLOduration=67.09127028 podStartE2EDuration="1m8.036462776s" totalImagesPullingTime="945.192496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:24.035399175 +0000 UTC m=+68.048670850" watchObservedRunningTime="2026-10-30 00:32:24.036462776 +0000 UTC m=+68.049734452" Oct 30 00:32:24.097428 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:24.097355 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:24.260036 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:24.259964 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:24.411747 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:24.411667 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:25.293565 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:25.293532 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:25.428909 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:25.428374 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:25.473236 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:25.473201 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:26.995747 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:26.995705 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4q86" event={"ID":"42079552-0ddd-4e83-85ef-604e435466eb","Type":"ContainerStarted","Data":"3361106aad498979fdf2448c1e576cda3645cbe79435f95c9b3bb5eb1571961e"} Oct 30 00:32:26.996203 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:26.995824 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:27.040884 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:27.040810 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f4q86" podStartSLOduration=67.930595845 podStartE2EDuration="1m11.040793688s" totalImagesPullingTime="3.110197843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:27.038949958 +0000 UTC m=+71.052221637" watchObservedRunningTime="2026-10-30 00:32:27.040793688 +0000 UTC m=+71.054065365" Oct 30 00:32:27.699305 ip-10-0-131-130 kubenswrapper[2577]: I1030 00:32:27.699134 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:32:27.699311 ip-10-0-131-130 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:32:27.707580 ip-10-0-131-130 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:32:27.707759 ip-10-0-131-130 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:32:27.707810 ip-10-0-131-130 systemd[1]: kubelet.service: Consumed 4.457s CPU time. Oct 30 00:32:27.727325 ip-10-0-131-130 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:32:27.790243 ip-10-0-131-130 kubenswrapper[5938]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:32:27.790243 ip-10-0-131-130 kubenswrapper[5938]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:32:27.790243 ip-10-0-131-130 kubenswrapper[5938]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:32:27.790243 ip-10-0-131-130 kubenswrapper[5938]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:32:27.792665 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792593 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.792665 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792660 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.792665 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792665 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.792665 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792669 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792673 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792678 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792682 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792686 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792689 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792693 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792697 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792701 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792705 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792709 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792713 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792717 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792721 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792725 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792729 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792734 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792738 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792742 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.793000 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792746 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792751 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792755 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792759 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792763 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792773 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792777 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792782 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792786 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792791 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792795 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792799 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792803 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792808 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792812 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792816 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792821 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792825 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792829 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792833 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.793822 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792837 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792841 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792846 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792850 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792855 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792876 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792881 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792886 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792890 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792896 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792901 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792906 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792910 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792914 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792918 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792923 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792927 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792931 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792936 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.794708 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792941 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792946 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792950 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792956 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792963 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792968 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792976 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792982 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792987 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792993 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.792997 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793001 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793005 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793010 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793016 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793020 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793024 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793028 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793033 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.795544 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793037 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793041 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793045 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793050 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793054 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793058 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793064 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793068 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793073 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793078 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793082 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793091 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793095 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793101 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793105 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793109 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793113 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793118 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793122 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793127 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.796418 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793131 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793135 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793141 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793146 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793150 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.793154 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794387 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794398 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794403 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794409 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794415 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794419 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794424 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794428 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794433 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794437 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794441 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794445 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794453 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.797348 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794457 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794461 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794466 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794470 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794476 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794481 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794485 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794490 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794495 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794500 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794504 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794508 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794513 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794517 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794521 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794525 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794530 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794534 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794538 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794543 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.798218 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794548 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794552 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794556 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794561 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794565 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794569 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794573 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794577 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794582 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794586 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794590 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794597 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794602 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794606 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794610 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794614 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794620 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794624 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794629 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794633 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.799100 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794639 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794646 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794653 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794659 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794664 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794669 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794673 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794678 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794682 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794686 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794690 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794694 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794699 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794703 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794707 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794710 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794714 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794718 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794722 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794726 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.799648 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794730 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794734 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794739 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794746 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794750 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794754 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794758 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794763 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794769 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794773 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794777 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794782 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794786 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794792 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794796 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794800 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794804 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794808 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794813 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.800190 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794817 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794822 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794826 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794830 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794834 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794839 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794843 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794847 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794851 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794857 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794877 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794882 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794886 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.794889 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795727 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795738 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795745 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795749 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795753 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795757 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.800710 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795762 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795767 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795771 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795776 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795780 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795784 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795790 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795795 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795799 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795804 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795808 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795812 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795817 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795821 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795825 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795830 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795834 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795838 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795842 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795847 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.801248 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795851 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795856 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795885 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795889 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795893 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795898 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795902 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795906 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795911 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795915 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795920 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795924 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795928 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795933 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795937 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795941 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795946 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795950 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795955 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795960 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.801721 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795964 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795969 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795973 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795977 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795982 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795986 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795990 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795994 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.795998 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796002 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796007 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796012 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796018 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796022 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796026 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796030 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796035 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796039 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796043 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.802211 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796047 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796052 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796057 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796061 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796066 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796070 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796074 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796079 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796083 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796087 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796091 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796095 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796100 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796116 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796120 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796124 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796129 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796135 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796141 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.802670 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796146 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796153 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796157 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796161 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796165 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796169 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796174 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796181 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796186 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796190 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796194 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796199 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796203 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796207 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796211 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796217 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796222 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796226 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796230 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796235 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.803160 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796239 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.796243 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796375 5938 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796386 5938 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796398 5938 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796404 5938 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796411 5938 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796420 5938 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796428 5938 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796434 5938 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796439 5938 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796443 5938 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796449 5938 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796455 5938 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796460 5938 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796464 5938 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796469 5938 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796474 5938 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796479 5938 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796486 5938 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796492 5938 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796497 5938 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796504 5938 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796510 5938 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:32:27.803857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796515 5938 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796521 5938 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796526 5938 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796531 5938 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796536 5938 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796542 5938 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796547 5938 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796552 5938 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796557 5938 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796562 5938 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796569 5938 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796573 5938 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796578 5938 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796583 5938 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796588 5938 flags.go:64] FLAG: --enable-server="true" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796592 5938 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796611 5938 flags.go:64] FLAG: --event-burst="100" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796616 5938 flags.go:64] FLAG: --event-qps="50" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796620 5938 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796626 5938 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796631 5938 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796637 5938 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796642 5938 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796647 5938 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796652 5938 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:32:27.804669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796656 5938 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796661 5938 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796668 5938 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796673 5938 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796678 5938 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796684 5938 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796689 5938 flags.go:64] FLAG: --feature-gates="" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796695 5938 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796700 5938 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796705 5938 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796710 5938 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796715 5938 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796721 5938 flags.go:64] FLAG: --help="false" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796725 5938 flags.go:64] FLAG: --hostname-override="ip-10-0-131-130.ec2.internal" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796731 5938 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796736 5938 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796741 5938 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796746 5938 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796751 5938 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796756 5938 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796761 5938 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796766 5938 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796795 5938 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:32:27.805358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796802 5938 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796816 5938 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796822 5938 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796827 5938 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796832 5938 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796837 5938 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796843 5938 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796847 5938 flags.go:64] FLAG: --lock-file="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796852 5938 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796857 5938 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796881 5938 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796892 5938 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796896 5938 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796901 5938 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796908 5938 flags.go:64] FLAG: --logging-format="text" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796912 5938 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796918 5938 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796922 5938 flags.go:64] FLAG: --manifest-url="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796927 5938 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796934 5938 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796939 5938 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796945 5938 flags.go:64] FLAG: --max-pods="110" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796950 5938 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796955 5938 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796960 5938 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:32:27.806018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796964 5938 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796969 5938 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796975 5938 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796980 5938 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796991 5938 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.796995 5938 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797000 5938 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797005 5938 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797010 5938 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797016 5938 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797021 5938 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797025 5938 flags.go:64] FLAG: --port="10250" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797030 5938 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797035 5938 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06cb5623e71db73ee" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797040 5938 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797045 5938 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797050 5938 flags.go:64] FLAG: --register-node="true" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797054 5938 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797062 5938 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797067 5938 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797072 5938 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797078 5938 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797084 5938 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797089 5938 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797094 5938 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797099 5938 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:32:27.806711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797104 5938 flags.go:64] FLAG: --runonce="false" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797108 5938 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797114 5938 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797119 5938 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797124 5938 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797128 5938 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797133 5938 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797138 5938 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797143 5938 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797148 5938 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797153 5938 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797157 5938 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797162 5938 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797167 5938 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797172 5938 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797177 5938 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797193 5938 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797198 5938 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797203 5938 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797213 5938 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797218 5938 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797222 5938 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797227 5938 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797232 5938 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797239 5938 flags.go:64] FLAG: --v="2" Oct 30 00:32:27.807366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797251 5938 flags.go:64] FLAG: --version="false" Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797257 5938 flags.go:64] FLAG: --vmodule="" Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797265 5938 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.797271 5938 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797529 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797537 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797542 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797556 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797561 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797565 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797570 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797574 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797579 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797583 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797587 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797592 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797596 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797600 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797605 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797609 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.808038 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797613 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797618 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797622 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797626 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797637 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797643 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797647 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797651 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797656 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797660 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797664 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797671 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797678 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797685 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797692 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797697 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797702 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797707 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797711 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.808742 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797716 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797720 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797724 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797729 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797733 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797738 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797742 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797747 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797751 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797755 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797759 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797764 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797768 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797772 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797777 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797782 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797786 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797790 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797802 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.809324 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797807 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797811 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797816 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797822 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797827 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797834 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797838 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797843 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797849 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797853 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797857 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797877 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797881 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797886 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797890 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797895 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797899 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797904 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797908 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797912 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.809771 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797917 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797922 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797926 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797930 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797934 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797960 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797966 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797971 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797976 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797981 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797986 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.797990 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798002 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798007 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798011 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798016 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798021 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798027 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798032 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798037 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.810280 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798042 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798046 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798051 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798055 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798059 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798064 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798068 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798072 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798077 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798081 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798086 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798090 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.798132 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798199 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798207 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798211 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.810909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798217 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798221 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798225 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798229 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798233 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798237 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798242 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798246 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798250 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798262 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798268 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798272 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798276 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798282 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798288 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798293 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798299 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798304 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798308 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.811382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798313 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798319 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798325 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798330 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798334 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798339 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798343 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798348 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798352 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798357 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798361 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798365 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798370 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798374 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798379 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798383 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798389 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798393 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798398 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798402 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.811892 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798406 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798410 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798414 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798426 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798431 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798435 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798440 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798446 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798451 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798456 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798460 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798465 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798469 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798474 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798478 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798482 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798487 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798492 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798496 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798500 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.812387 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798504 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798508 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798513 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798517 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798522 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798526 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798531 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798535 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798539 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798543 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798547 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798551 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798555 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798560 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798564 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798569 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798583 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798587 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798591 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798600 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.812909 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798604 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798611 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798615 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798619 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798624 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798628 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798632 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798636 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798640 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798645 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798649 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798653 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798658 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798662 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798666 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798670 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798675 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798679 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.813396 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798687 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798694 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798699 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798703 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798708 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.798712 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.802225 5938 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.802241 5938 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.802259 5938 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.802263 5938 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802321 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802326 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802330 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802333 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802336 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802339 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802342 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802345 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802348 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802351 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.813829 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802353 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802356 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802359 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802361 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802364 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802366 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802369 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802372 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802375 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802377 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802380 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802383 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802385 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802388 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802391 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802394 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802397 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802399 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802404 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802408 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802411 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802415 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802417 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802420 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802423 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802425 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802429 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802432 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802434 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802437 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802440 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802443 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802446 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802449 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802452 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802454 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802457 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802460 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.814826 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802462 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802466 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802468 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802471 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802473 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802476 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802479 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802481 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802484 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802486 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802490 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802494 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802497 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802500 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802503 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802506 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802509 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802511 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802514 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802516 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.815313 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802520 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802522 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802525 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802528 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802530 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802534 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802537 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802540 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802542 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802545 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802547 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802550 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802552 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802555 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802558 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802560 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802563 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802566 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802568 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802571 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.815796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802573 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802575 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802578 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802581 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802583 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802586 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802588 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802591 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802593 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802596 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802598 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802601 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802604 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802607 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802609 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802612 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802614 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.816358 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802617 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.802644 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802810 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802849 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802855 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802922 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802929 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802936 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802943 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802949 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802954 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802960 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802965 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802970 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802982 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802987 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.816796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802992 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.802997 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803002 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803007 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803012 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803017 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803022 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803027 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803032 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803037 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803043 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803054 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803059 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803063 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803068 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803073 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803078 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803083 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803088 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.817203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803105 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803110 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803115 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803120 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803130 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803135 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803140 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803146 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803150 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803155 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803160 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803165 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803170 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803175 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803180 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803185 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803190 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803200 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803205 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803210 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.817658 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803215 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803220 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803225 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803230 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803235 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803240 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803244 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803249 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803254 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803259 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803268 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803273 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803277 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803286 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803293 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803299 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803304 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803309 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803314 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.818210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803319 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803324 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803329 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803335 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803345 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803350 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803355 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803360 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803365 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803369 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803375 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803380 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803385 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803390 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803395 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803400 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803410 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803415 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803420 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.818661 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803426 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803431 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803436 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803441 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803446 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803451 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803455 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803461 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803467 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803472 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803481 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803487 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803492 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803497 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803502 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.803562 5938 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:32:27.819203 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803668 5938 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803675 5938 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803680 5938 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803694 5938 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803705 5938 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803711 5938 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803716 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803721 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803725 5938 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803730 5938 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803735 5938 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803740 5938 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803744 5938 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803749 5938 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803754 5938 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803758 5938 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803763 5938 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803773 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803778 5938 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803783 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:32:27.819601 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803787 5938 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803792 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803797 5938 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803802 5938 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803807 5938 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803815 5938 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803823 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803828 5938 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803833 5938 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803843 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803848 5938 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803852 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803857 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803880 5938 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803885 5938 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803890 5938 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803895 5938 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803900 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803909 5938 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:32:27.820116 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803912 5938 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803915 5938 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803921 5938 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803924 5938 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803927 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803932 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803934 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803937 5938 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803941 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.803946 5938 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804137 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804147 5938 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804150 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804154 5938 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804158 5938 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804161 5938 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804163 5938 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804166 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804169 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:32:27.820604 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804173 5938 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804176 5938 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804179 5938 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804183 5938 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804186 5938 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804189 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804192 5938 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804195 5938 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804197 5938 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804200 5938 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804203 5938 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804206 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804209 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804212 5938 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804215 5938 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804218 5938 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804220 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804224 5938 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804226 5938 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804229 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:32:27.821102 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804232 5938 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804234 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804237 5938 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804240 5938 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804243 5938 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804245 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804248 5938 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804251 5938 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804255 5938 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804260 5938 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804264 5938 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804267 5938 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804270 5938 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804273 5938 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804276 5938 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804279 5938 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804281 5938 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804284 5938 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804287 5938 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:32:27.821790 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804290 5938 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804292 5938 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804295 5938 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804298 5938 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804300 5938 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804303 5938 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804306 5938 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804309 5938 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:27.804313 5938 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.804397 5938 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805327 5938 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805388 5938 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805557 5938 server.go:1014] "Starting client certificate rotation" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805694 5938 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805736 5938 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:16 +0000 UTC" deadline="2028-06-13 05:55:43.227618831 +0000 UTC" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.805762 5938 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14213h23m15.421858862s" Oct 30 00:32:27.822561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.806051 5938 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:32:27.823163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.807174 5938 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:32:27.823163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.808403 5938 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:32:27.823163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.810474 5938 log.go:25] "Validated CRI v1 image API" Oct 30 00:32:27.823163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.811169 5938 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:32:27.823163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.813328 5938 fs.go:135] Filesystem UUIDs: map[4bb1c90b-3da9-4139-9985-801e8fae73b9:/dev/nvme0n1p3 55befcf6-b936-4fa1-a3a9-2e4f9f2b62dc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 00:32:27.823431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.813347 5938 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6/userdata/shm major:0 minor:275 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0562361d066cf1a82d021d9a41c9bfdbd4181c8717b7bef724ec3eecaea30a0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0562361d066cf1a82d021d9a41c9bfdbd4181c8717b7bef724ec3eecaea30a0e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46fe183b7123e4c66e570906720c9e0266ff2bef4b8f505f9b59dd5cddcf5bc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46fe183b7123e4c66e570906720c9e0266ff2bef4b8f505f9b59dd5cddcf5bc2/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6/userdata/shm major:0 minor:484 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e35688a19a204bedae8fdeab3162e4e319d7fe5b161ee7e99aebd7d81c8f801f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e35688a19a204bedae8fdeab3162e4e319d7fe5b161ee7e99aebd7d81c8f801f/userdata/shm major:0 minor:427 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/config-reloader/2 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/prometheus/15 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~empty-dir/config-out major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/kube-api-access-72gs7:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/kube-api-access-72gs7 major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/tls-assets major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:417 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/web-config major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~projected/kube-api-access-lmlsr:{mountpoint:/var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~projected/kube-api-access-lmlsr major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~projected/kube-api-access-brxm8:{mountpoint:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~projected/kube-api-access-brxm8 major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24461adc-8328-4dd9-a4ac-859666bdc68c/volumes/kubernetes.io~projected/kube-api-access-bhj5c:{mountpoint:/var/lib/kubelet/pods/24461adc-8328-4dd9-a4ac-859666bdc68c/volumes/kubernetes.io~projected/kube-api-access-bhj5c major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34aaeeff-dda3-47f1-84cf-ea29bb1bd772/volumes/kubernetes.io~projected/kube-api-access-nrbbs:{mountpoint:/var/lib/kubelet/pods/34aaeeff-dda3-47f1-84cf-ea29bb1bd772/volumes/kubernetes.io~projected/kube-api-access-nrbbs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~projected/kube-api-access-ls9rr:{mountpoint:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~projected/kube-api-access-ls9rr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~projected/kube-api-access-k5jrm:{mountpoint:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~projected/kube-api-access-k5jrm major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42079552-0ddd-4e83-85ef-604e435466eb/volumes/kubernetes.io~projected/kube-api-access-kp9tb:{mountpoint:/var/lib/kubelet/pods/42079552-0ddd-4e83-85ef-604e435466eb/volumes/kubernetes.io~projected/kube-api-access-kp9tb major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~projected/kube-api-access-bjgqb:{mountpoint:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~projected/kube-api-access-bjgqb major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56772ce8-a261-4951-b032-2b02e0f486ec/volumes/kubernetes.io~projected/kube-api-access-4pqc5:{mountpoint:/var/lib/kubelet/pods/56772ce8-a261-4951-b032-2b02e0f486ec/volumes/kubernetes.io~projected/kube-api-access-4pqc5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~projected/kube-api-access-ztkgx:{mountpoint:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~projected/kube-api-access-ztkgx major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~projected/kube-api-access-tnslc:{mountpoint:/var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~projected/kube-api-access-tnslc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~secret/metrics-certs major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~projected/kube-api-access-t9bdb:{mountpoint:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~projected/kube-api-access-t9bdb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/alertmanager/9 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/config-reloader/6 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~empty-dir/config-out major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/kube-api-access-fb89b:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/kube-api-access-fb89b major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/tls-assets major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/config-volume major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/web-config major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ed2348-6522-439c-9381-92240fabb46d/volumes/kubernetes.io~projected/kube-api-access-5tzgc:{mountpoint:/var/lib/kubelet/pods/73ed2348-6522-439c-9381-92240fabb46d/volumes/kubernetes.io~projected/kube-api-access-5tzgc major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~projected/kube-api-access-rbw2w:{mountpoint:/var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~projected/kube-api-access-rbw2w major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79358397-9466-4f7b-9349-cd976b869ead/volumes/kubernetes.io~projected/kube-api-access-bgjzq:{mountpoint:/var/lib/kubelet/pods/79358397-9466-4f7b-9349-cd976b869ead/volumes/kubernetes.io~projected/kube-api-access-bgjzq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~projected/kube-api-access-dpbjc:{mountpoint:/var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~projected/kube-api-access-dpbjc major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~projected/kube-api-access-x4859:{mountpoint:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~projected/kube-api-access-x4859 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e691205-7020-4ddd-9fe6-61bb9d460d30/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9e691205-7020-4ddd-9fe6-61bb9d460d30/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:476 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b8acd5-22da-46b5-a40d-7e45cafcc342/volumes/kubernetes.io~projected/kube-api-access-qktzs:{mountpoint:/var/lib/kubelet/pods/c2b8acd5-22da-46b5-a40d-7e45cafcc342/volumes/kubernetes.io~projected/kube-api-access-qktzs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~projected/kube-api-access-bz9pt:{mountpoint:/var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~projected/kube-api-access-bz9pt major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~projected/kube-api-access-6qfd7:{mountpoint:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~projected/kube-api-access-6qfd7 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~projected/kube-api-access-n2gsf:{mountpoint:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~projected/kube-api-access-n2gsf major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:272 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c1c69b681d3590dc36bfcd89fe7222dedae6031f2f4eb08a83f70cd2ee52832f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/898c98479378715a26538fa4e7baf7fb4642ac10c6657c3b1ef47c91e7b200bc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9f3bceec7654a3d57b83e2ed33b4a437848cb13e8a7e197aa7b2c70b1efc643b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/447afdc2a441c7b64fc415847db87609312482750a431ff3ffe0d4ee1f63ee06/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c1568d0317a68ef5dd397d3d14c02be7d959fe97a27790cb7fe79634ed7ad119/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b888e776f0a1785b9de3d40bd65ce2edc83f94a9381a24e038c24a58eb9d2192/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/baee42ea15ba7e673c0aacb7665b8f5ad170f89faa69daebe9cc6e79eec528aa/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7525bd8e26785732cfcf3265a97925046a34badd568a03d584d7ee67c3364507/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a852dff50ce358a586cab20e99acede5c17b6c6a493fb4fdb09120bf5d37f55e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ba1f82535c934124fa8833047d9368dc07b6f9b1b726ab22ca1f1483235b1db2/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/b771196e33811b15afbda61f834540c01ac7b1f15b462f2d4183159c9381ff13/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/bca77b4e69bdbd307e080c5f4fd802166a5b99a64b744c59add17e71825970b2/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/114e1736f05d3a751197852ce314960c3a8c1d6b29535247b5c91143b81d2884/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/eb6567f9661fa01c14da218f9804564e884210e84cec4eeab6aab1daef87cca1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/065ada26796982d6088091574b2663d8423229e3c5d57aca7bffb8572b5efd66/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/768730c945bc97696b1bd014311e94c5b122d65ce3ecfad56228ce98d11b471a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1aabc43495de1bac9e0d8496cfb19c9f02aa26fc80f110b2c73843d53bbc3881/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/524a53edc18137e080ebe38e2e5012860d23f6cad3a83d8b77c1d5c2d5f6b123/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c0d33f233ed09a1295f2af597773205e98bb5a72d9a5e63f0fcef62fae1488be/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0aad4829192d14a630579593dccb901ac6e9a3a5db913d7215cb0ed3d9e32148/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b440c717922cafbd8f826d3b59d08c665322bd4cf4d52a6967cf46282187f94d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5ae06fb7c0b76e827779898ba6f5c8a4b432ef184350874fe39d329df596cf17/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a6a17548ccd662c00399e09b6da6acec8829bbee3602d964adfe7cb12bef69fb/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/b54873fda2d50eff4740176ab507d1699c9ddd95d8d06c6cf2b80c0a19a965fe/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2489f529a4786d7b721c462ab0f74ab830eb1cef0120dc1e822bf4555773d147/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/6749d6cde27c440422371d5015d1d0a35bf52c0afe0e55272b287690a20408a6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/63ff6ba994dc8e14a11e81a14bc804c1890b825312ddc566ce7a695a1bc6f904/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/e13a0b2c0db0c6da1197ce19134e70f631489e6fdec8e0c9ec02f705e07f335e/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/f995cd8e7cbd9fa21debba298dee4239d76d6d00e54343a407794ac5a69b0c40/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/78153f86849f4f897de532f7657d61ba86875014ccc62761c666b8a47b3df1af/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/7132698079a004b95149db570d6a26583856db7a72a428283f28d8d9b9bc0f90/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/a3607d81dcec80a3750540fbb59ac285acae32e4e142f00480cb089a8558b5cf/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/55703a947df7670071f8db9dba74b22543ad10acf55114badabf5f45e13e7bc3/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/1984baea4dca657342ce95b4012d6524e68e7fbe632843aa052db59428e7b7fa/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/b3f69b1877c42a995dfa82ee0d0c912b3511b8dc48d7ede63a9953577fdb673e/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/a0191132b9102b109f8fffa77037dcdfdbec6013d060912e9087848c2d5ce2b8/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/5b4251a21a260f5d759eee5f525bf327e0ef94cc151621260b97b15b265b1251/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/a9baa3401428504f81b85a4993538bb400e1773ffb0b038e422d5cbe92c128ac/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/d5e3a9a0d4624795f8f1bcbd222594f99e8add502de7239d75c885d3bf23ce74/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/ab468f336c8c9f94e777b82039d177b61d76622e9ca437aa92447006956e71e0/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/c46733b686f0d5fd903490bef5c7301348797e06cb7930a332949874365df18e/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/f8e6dd0c8d78fc8ddd2b275f7f06df37b8edccdd8aad83a2322a6c39b2048c22/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/75393027c51fb9f12391914bdc4e0c740818112ce52b338679d7bd2e1dbcf14f/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e7286983c539ca2bb79ad28bffcf0b225913c3c5b229e063a40d212de4349325/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/8cd60fbccd7da531be0e1d114458f826d3f44e9f267fc45b6e21493fe8970a8d/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/ccb20021d12fd110af17adc43addd66ac1e74d8f04842cd40792d432d9f35e80/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/3fe0e7acdf7bd2c87bf05cbd3503b639329f5fae41eb6150ae2bdcf332327cda/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/67630b7bfd4582ded53917867aaaff7074adb5e64f3dd6af451fa30c3d85a8c1/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/057659e95fb3d8cd6942198bee8835c71cd4e52e8f9b1338b7f909da6a8824f6/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/57f30c90556a18778d9bcc891188c56b85df9d7041cafd5e7ad352a68c8bfde9/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/3599a1d562fd9ed21504d369dc683afabd291895c8b94830cec19e81ebbcbca9/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/d014e72c1d78ee96384b2a83359c85a831d255f2bcee9c4b1b0e7f5304eb2cd7/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/9ccb80f0fdc5d8c054f224dd5fef4185967f0f0f735f80df6a4ab7821f5fe858/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/be8d0b52ee33b39d0da6ce53526dfee6b290334215c020ed9943e249e7ee2cab/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/6554fcf783a2f4291f16f237ec76ef3c3313296f44fe9a49638f311b5997368b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/918d6b51571ce1f6c168e52fb2214c09ddab86df9e1105c37a5e3c50b7f5aff5/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/3ce932f748ac944f647904f72348be0f6cc307e864e9e79e1803236a2d605c05/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/2991f0a7e7261f60ffe749ee5d8a6312a57d5ceaa099b5f9fc47c66df37f9fc3/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/51ebbea28687696cb794208aad7620358692aada9483e4dace24bd0409153716/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/c4b5fde41a78771fdf0091fc35a86511e394b463b16e11d9194b7785e389eb41/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/004ad539edcd4daeede8769856bd604324c05ce3249f7e9f9aaa186c2bfc98bd/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/c77e8b66f70e44dc964d16992d14e1ea2397265b903e1554466b5629424a9d82/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/5d3fc29198ebc185fc24723e1e73e1dfdb79612adebfe1060c823174e02e0461/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/c6312eae3eeda857da6590918bc0b1fd5a5fe1311730bbae79cedf863f6cc9cc/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/a62784116f4573c4f5bc97b475e61b3f6dfea771f591084951fcbe8e89e954e5/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/4b903517ddb8bd19ec81251f2af77b828f30e9cd5f1bdac4b4ff5cbb7d67e471/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/dfd51abb813c9f296f32369b1d6c96a4ac56265ddcfce173bd079260cbf339da/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/f4181c2fb15852926a111b511f51c377810286388502e8b03e9f57b6ed8215c6/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/50521aec75fc3c2cee1726cf0f18d3ca3485af9420bf484e5737f9244fd8c0d8/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/9fcbff4f1e64abcec152037c2d3027ea4f4145949c97308debad646881dfcf3e/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/c5d04396586252d6adb72c01bf44294610c2b2b4fab40be45f87e744b2df854d/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/a9edda92f797d837394117c533e872e0147db88785b10b618e028b0e3012ada0/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/3d6ab200f566717390abf0395e8adce321a16e882b507f6b3dcd8f32eab02e92/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0f6022eba3dc8ad3d4040b9ef438b684717697e3bb1be912b65bc755c238ecb2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/72b28bda7506dbf8e77e9042c5910ea08a7a79384470f5a268a02d9a214be3bd/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c37e1c16426d3c6de9e25fed2a438e246ea4718c567e1a8c42951de580133c34/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fab9f83a739fb3584734868daf7c9a947ff0a086d005d6c57d998876a9884bca/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4db48d61a7a6110105cad19b2fc0560de420389f0c1e2841e319e8111c2b9c7f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7d73ec4232aecf7c6a5b6527664ce51e8087ca3602e35ab9b28fbfba15ea1784/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2eb2303112678b2407f84defe5a36e3c98fe37f43dc8e9e4a01c083d33cc24e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a3e1da2366380ebabc1d889efae11ff483d80999934eaba4b7b6d16305f5e6d1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5565b602f6e946b4904e024f2028580740dbfae64938ea9afe07713921609ab8/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7c083a0fa1ed7ff0b3a156fbcdc5194d4b775d369910866bd1961e10e985b0a7/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.822153 5938 manager.go:217] Machine: {Timestamp:2026-10-30 00:32:27.821210775 +0000 UTC m=+0.061445879 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103866 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214253f1e7e19900f9aaaaf9787c71 SystemUUID:ec214253-f1e7-e199-00f9-aaaaf9787c71 BootID:dbedf14c-fe03-4ec6-b35e-8a18cebb67d6 Filesystems:[{Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e691205-7020-4ddd-9fe6-61bb9d460d30/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:476 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7827e0ae3f407c0e197e31e0181b26417f91c0c074e90bd436b16e4049675638/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46fe183b7123e4c66e570906720c9e0266ff2bef4b8f505f9b59dd5cddcf5bc2/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:364 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:424 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~projected/kube-api-access-n2gsf DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42079552-0ddd-4e83-85ef-604e435466eb/volumes/kubernetes.io~projected/kube-api-access-kp9tb DeviceMajor:0 DeviceMinor:479 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73ed2348-6522-439c-9381-92240fabb46d/volumes/kubernetes.io~projected/kube-api-access-5tzgc DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~projected/kube-api-access-6qfd7 DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~projected/kube-api-access-x4859 DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:422 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~projected/kube-api-access-dpbjc DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79358397-9466-4f7b-9349-cd976b869ead/volumes/kubernetes.io~projected/kube-api-access-bgjzq DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:417 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:416 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:425 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7fad984ae97274b2dec5ec1d0c92c52b1a0ffe8b2b382d4f370c2c934d057b1/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1de762273b6771d660e36cb869c716b7fbe7a037acefb4b6b937dc3e45f17642/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24461adc-8328-4dd9-a4ac-859666bdc68c/volumes/kubernetes.io~projected/kube-api-access-bhj5c DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8845be9fefbd972b9f00bcf5d3bdd65a9892771a6f32441e9343a396f322bcf6/userdata/shm DeviceMajor:0 DeviceMinor:484 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf4ef65455be54445f1e65e4092990fedaf7bb20fbcd70c2552cccb4ebb288a4/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a19d854e47e1bf3e78110d8153d0d8db6f5aeb332e9912b0e0d7f33f672cd5b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56772ce8-a261-4951-b032-2b02e0f486ec/volumes/kubernetes.io~projected/kube-api-access-4pqc5 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~projected/kube-api-access-bz9pt DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7128c39597a824f49586c2b65cda1b4b3f4821348d3515865efff1d9f0e63d59/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9149ff3-efaf-4600-b5fe-7699ffcd8db5/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~projected/kube-api-access-brxm8 DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31e5cd4dbb17e71391d7da8ae2dad08effc4c17a900ac43c6da9fb1a1b6aba66/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5aa2f9f5c08f4009ddc9be7cc3fa05faa02091771c2ba6ecc28c0d1b215f7d4/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7876ee6c-178d-4ca6-923a-d76c0867bd85/volumes/kubernetes.io~projected/kube-api-access-rbw2w DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b084828bd532de0bced98c454e82fc0bafe279bd4fc4f76c43a90bf38679b88/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b8acd5-22da-46b5-a40d-7e45cafcc342/volumes/kubernetes.io~projected/kube-api-access-qktzs DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0a3f8721fdb496efc3e2a44c68fd2f20bc4b6185f1f5550227865caf6ca750d/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:419 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:439 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04a254f7386b2d3384d05650bf46fa2fc0f871864ffcd76cac13ce29c7aa9ad6/userdata/shm DeviceMajor:0 DeviceMinor:275 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:472 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08604272-a8a8-44c9-bcc1-9b3f12a37d77/volumes/kubernetes.io~projected/kube-api-access-lmlsr DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4ed79ad-9a13-4899-8307-eff21d88d3e6/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2418bd154938c2810d7c5f01b6e93614cd0ef3aa356191a248dc670d62778e61/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0562361d066cf1a82d021d9a41c9bfdbd4181c8717b7bef724ec3eecaea30a0e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e35688a19a204bedae8fdeab3162e4e319d7fe5b161ee7e99aebd7d81c8f801f/userdata/shm DeviceMajor:0 DeviceMinor:427 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:443 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c099935a136ada69f5c7984c6d5b7b64d81ac3c8153594420c464916acc396cc/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34aaeeff-dda3-47f1-84cf-ea29bb1bd772/volumes/kubernetes.io~projected/kube-api-access-nrbbs DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bf30350-e3f7-4edd-a691-b499f83bccb6/volumes/kubernetes.io~projected/kube-api-access-t9bdb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6283ace2-25a6-43fe-b12c-1d952c5c95da/volumes/kubernetes.io~projected/kube-api-access-ztkgx DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/661abd2b-095c-4b75-ae58-a5c2a82ad0ff/volumes/kubernetes.io~projected/kube-api-access-tnslc DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3757f105-2260-48d2-be57-41ba8481ae63/volumes/kubernetes.io~projected/kube-api-access-k5jrm DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:411 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes/kubernetes.io~projected/kube-api-access-72gs7 DeviceMajor:0 DeviceMinor:426 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79858a62-0cea-444e-a2ce-fec46d542e55/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91aa19566693eedbda35f78d68cfe09855276af4982d2532b02692a82e591b9f/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes/kubernetes.io~projected/kube-api-access-bjgqb DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/kube-api-access-fb89b DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc63ec89801124cdae2cd018f2e81a130f4e1f9dfa511eb32a591e12849d8232/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf629046483195014212e8f8202f4751cdad4fb1b6b59c324575867325a2d48/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34e4db94-de44-4a89-9bfe-5ac1d5997038/volumes/kubernetes.io~projected/kube-api-access-ls9rr DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52c42e1a7185e11703518dc77cf8bc42f24493b36a56cfd0e6dedd5eedd6f2ff/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7003a5d-b3e4-4c2e-88d2-f362a5f78180/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04a254f7386b2d3 MacAddress:1a:00:bc:f5:5f:e3 Speed:10000 Mtu:8901} {Name:1de762273b6771d MacAddress:46:07:35:de:cf:a8 Speed:10000 Mtu:8901} {Name:45001f8bef284a3 MacAddress:12:88:c1:a8:36:95 Speed:10000 Mtu:8901} {Name:46fe183b7123e4c MacAddress:ca:02:f0:99:2e:81 Speed:10000 Mtu:8901} {Name:6b084828bd532de MacAddress:86:b8:67:92:8b:8b Speed:10000 Mtu:8901} {Name:7128c39597a824f MacAddress:a2:6d:9d:5f:4b:fe Speed:10000 Mtu:8901} {Name:7827e0ae3f407c0 MacAddress:4a:f2:fe:02:8d:7f Speed:10000 Mtu:8901} {Name:8845be9fefbd972 MacAddress:ee:99:2e:5b:4e:75 Speed:10000 Mtu:8901} {Name:a5aa2f9f5c08f40 MacAddress:5e:f1:4e:07:b4:90 Speed:10000 Mtu:8901} {Name:bcbce8fd9737935 MacAddress:b2:61:98:10:00:7a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c6:75:f3:83 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:da:e1:6a:f0:ec Speed:0 Mtu:8901} {Name:d70f4d71579684f MacAddress:ae:75:9f:15:9e:23 Speed:10000 Mtu:8901} {Name:dbf629046483195 MacAddress:d2:30:6b:be:96:98 Speed:10000 Mtu:8901} {Name:dc63ec89801124c MacAddress:16:1d:90:0a:bb:64 Speed:10000 Mtu:8901} {Name:e35688a19a204be MacAddress:4e:b5:f5:66:3f:2b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c6:75:f3:83 Speed:0 Mtu:9001} {Name:f0a3f8721fdb496 MacAddress:be:9e:36:29:99:df Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0a:63:43:6b:cf:c9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:5f:71:16:55:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823007 5938 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823056 5938 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823399 5938 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823431 5938 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823607 5938 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823619 5938 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823632 5938 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823649 5938 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823747 5938 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:32:27.824050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.823986 5938 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824115 5938 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824156 5938 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824174 5938 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824187 5938 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824199 5938 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:32:27.824918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.824210 5938 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:32:27.825214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.825198 5938 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:32:27.825272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.825214 5938 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:32:27.825691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.825675 5938 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:32:27.828315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828301 5938 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:32:27.828315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828314 5938 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:32:27.828411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828395 5938 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:32:27.828563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828552 5938 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:32:27.828594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828581 5938 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:32:27.828671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828662 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828675 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828682 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828687 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828693 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828699 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828704 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:32:27.828711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828710 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828722 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828728 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828736 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828744 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828770 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:32:27.829002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.828775 5938 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:32:27.831825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.831812 5938 server.go:1263] "Started kubelet" Oct 30 00:32:27.832005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.831935 5938 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:32:27.832420 ip-10-0-131-130 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:32:27.832597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.831919 5938 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:32:27.832597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.832481 5938 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:32:27.832889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.832831 5938 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:32:27.833209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.833176 5938 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:32:27.839057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839038 5938 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839061 5938 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839120 5938 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:18 +0000 UTC" deadline="2028-04-13 07:27:50.004353696 +0000 UTC" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839151 5938 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12750h55m22.165206104s" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839151 5938 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839170 5938 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:27.839166 5938 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-130.ec2.internal\" not found" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839162 5938 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839962 5938 factory.go:55] Registering systemd factory Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.839982 5938 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840350 5938 factory.go:153] Registering CRI-O factory Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840364 5938 factory.go:223] Registration of the crio container factory successfully Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840411 5938 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840431 5938 factory.go:103] Registering Raw factory Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840449 5938 manager.go:1199] Started watching for new ooms in manager Oct 30 00:32:27.840817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.840782 5938 manager.go:319] Starting recovery of all containers Oct 30 00:32:27.842610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842570 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" volumeName="kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc" seLinuxMountContext="" Oct 30 00:32:27.842695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842612 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc" volumeName="kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 00:32:27.842695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842630 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7003a5d-b3e4-4c2e-88d2-f362a5f78180" volumeName="kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt" seLinuxMountContext="" Oct 30 00:32:27.842695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842645 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.842695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842670 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08604272-a8a8-44c9-bcc1-9b3f12a37d77" volumeName="kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume" seLinuxMountContext="" Oct 30 00:32:27.842695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842683 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08604272-a8a8-44c9-bcc1-9b3f12a37d77" volumeName="kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842697 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842732 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842749 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79358397-9466-4f7b-9349-cd976b869ead" volumeName="kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842763 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9149ff3-efaf-4600-b5fe-7699ffcd8db5" volumeName="kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842775 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bf30350-e3f7-4edd-a691-b499f83bccb6" volumeName="kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842788 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79858a62-0cea-444e-a2ce-fec46d542e55" volumeName="kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842801 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842815 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842831 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert" seLinuxMountContext="" Oct 30 00:32:27.842845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842845 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842858 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4db94-de44-4a89-9bfe-5ac1d5997038" volumeName="kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842888 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842903 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842916 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc" volumeName="kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842945 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e691205-7020-4ddd-9fe6-61bb9d460d30" volumeName="kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842958 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42079552-0ddd-4e83-85ef-604e435466eb" volumeName="kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842971 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="661abd2b-095c-4b75-ae58-a5c2a82ad0ff" volumeName="kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842984 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.842999 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7876ee6c-178d-4ca6-923a-d76c0867bd85" volumeName="kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843012 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843024 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843037 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08604272-a8a8-44c9-bcc1-9b3f12a37d77" volumeName="kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843068 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843081 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4db94-de44-4a89-9bfe-5ac1d5997038" volumeName="kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843094 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 00:32:27.843125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843107 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843130 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08604272-a8a8-44c9-bcc1-9b3f12a37d77" volumeName="kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843143 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843157 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843171 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843184 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc" volumeName="kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843196 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4db94-de44-4a89-9bfe-5ac1d5997038" volumeName="kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843209 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843224 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843237 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843251 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843265 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843279 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843290 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56772ce8-a261-4951-b032-2b02e0f486ec" volumeName="kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843304 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843317 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843330 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843345 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843357 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843372 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc" volumeName="kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843385 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843397 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843410 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9149ff3-efaf-4600-b5fe-7699ffcd8db5" volumeName="kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843425 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843438 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aaeeff-dda3-47f1-84cf-ea29bb1bd772" volumeName="kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843459 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843473 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aaeeff-dda3-47f1-84cf-ea29bb1bd772" volumeName="kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843487 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4db94-de44-4a89-9bfe-5ac1d5997038" volumeName="kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843498 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e4db94-de44-4a89-9bfe-5ac1d5997038" volumeName="kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843512 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843528 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.843542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843542 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843554 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843566 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843579 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843601 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79858a62-0cea-444e-a2ce-fec46d542e55" volumeName="kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843614 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843626 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843638 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843651 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc" volumeName="kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843663 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843676 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843698 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b8acd5-22da-46b5-a40d-7e45cafcc342" volumeName="kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843712 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843724 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" volumeName="kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843736 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ed2348-6522-439c-9381-92240fabb46d" volumeName="kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843749 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9149ff3-efaf-4600-b5fe-7699ffcd8db5" volumeName="kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843764 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843777 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843790 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843803 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843817 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843829 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7003a5d-b3e4-4c2e-88d2-f362a5f78180" volumeName="kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843842 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9149ff3-efaf-4600-b5fe-7699ffcd8db5" volumeName="kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843855 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843886 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc" volumeName="kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843898 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24461adc-8328-4dd9-a4ac-859666bdc68c" volumeName="kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843915 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843927 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56772ce8-a261-4951-b032-2b02e0f486ec" volumeName="kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843940 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b8acd5-22da-46b5-a40d-7e45cafcc342" volumeName="kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843953 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843966 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54406dd0-d002-4fc9-84b4-41532b139761" volumeName="kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843979 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ed2348-6522-439c-9381-92240fabb46d" volumeName="kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.843996 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844010 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844024 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b8acd5-22da-46b5-a40d-7e45cafcc342" volumeName="kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844036 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4ed79ad-9a13-4899-8307-eff21d88d3e6" volumeName="kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844049 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3757f105-2260-48d2-be57-41ba8481ae63" volumeName="kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844063 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bf30350-e3f7-4edd-a691-b499f83bccb6" volumeName="kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844076 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79358397-9466-4f7b-9349-cd976b869ead" volumeName="kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844087 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844101 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79858a62-0cea-444e-a2ce-fec46d542e55" volumeName="kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844115 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b8acd5-22da-46b5-a40d-7e45cafcc342" volumeName="kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844129 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844142 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844154 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844169 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844185 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" volumeName="kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844200 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844213 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bf30350-e3f7-4edd-a691-b499f83bccb6" volumeName="kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844226 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7876ee6c-178d-4ca6-923a-d76c0867bd85" volumeName="kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844273 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7063ed5c-d12e-4887-812f-4e1eeea42495" volumeName="kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844286 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79358397-9466-4f7b-9349-cd976b869ead" volumeName="kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844302 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6283ace2-25a6-43fe-b12c-1d952c5c95da" volumeName="kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844314 5938 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79858a62-0cea-444e-a2ce-fec46d542e55" volumeName="kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc" seLinuxMountContext="" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844326 5938 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:32:27.844418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.844335 5938 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:32:27.866582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.866414 5938 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:32:27.868039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.868012 5938 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:32:27.868039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.868037 5938 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:32:27.868197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.868055 5938 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:32:27.868197 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:27.868091 5938 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:32:27.874400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.874380 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:32:27.875116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.875077 5938 generic.go:356] "Generic (PLEG): container finished" podID="5b21b9005987db40e0d6504b950f5deb" containerID="137309faba7fea05098e6d9415726070ef5c0692681dead6b25c0fc1ddf2e974" exitCode=0 Oct 30 00:32:27.875194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.875129 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:32:27.884157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.884132 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="72e98b4e42c17edf9f57b20b784ea8c84a222392e3cad7636cc6ee0b10458141" exitCode=0 Oct 30 00:32:27.884747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.884642 5938 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:32:27.884747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.884656 5938 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:32:27.894525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.894499 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:32:27.894941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.894921 5938 generic.go:356] "Generic (PLEG): container finished" podID="34e4db94-de44-4a89-9bfe-5ac1d5997038" containerID="f4dade5736965cb021327e2a9d58aa56910319b4af396024c9db38277c76edeb" exitCode=1 Oct 30 00:32:27.897423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.897394 5938 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:32:27.907372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.907351 5938 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:32:27.908178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.908158 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039" exitCode=0 Oct 30 00:32:27.908262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.908214 5938 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:32:27.914799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914778 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="e02b6dc23274bd31681f6ad62df106c2f10369e167774d07d9221d5d9ec6f9c6" exitCode=0 Oct 30 00:32:27.914799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914796 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="c205dbc6eecb34c0cae5a17334371ccfa539be4f8f64c268235f1896e6fd7c53" exitCode=0 Oct 30 00:32:27.914944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914804 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="8fa833d8d05260462b1149ad2cfcd47116aa30ab87d960024f5fac091aafa7c0" exitCode=0 Oct 30 00:32:27.914944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914810 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="e5a47156cce88cf731a510f33c3e0418ad177b444dccf06842bee925ad97d617" exitCode=0 Oct 30 00:32:27.914944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914816 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="a32c12d1ef59feffedef0d86da87674f447879d8de411e0378b73b267c6bb697" exitCode=0 Oct 30 00:32:27.914944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.914821 5938 generic.go:356] "Generic (PLEG): container finished" podID="c2b8acd5-22da-46b5-a40d-7e45cafcc342" containerID="ca738f142e06e179ed6c54b631ef96c25df8f9efa40dadd126e1033cef678447" exitCode=0 Oct 30 00:32:27.916911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.916892 5938 generic.go:356] "Generic (PLEG): container finished" podID="6283ace2-25a6-43fe-b12c-1d952c5c95da" containerID="38b9766e6dd7fc88e7eb9e8c485a2b3412824e4c2808ae40f128cd420d90d0c4" exitCode=0 Oct 30 00:32:27.962908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.962826 5938 manager.go:324] Recovery completed Oct 30 00:32:27.968164 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:27.968140 5938 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 00:32:27.991114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991085 5938 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:32:27.991114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991103 5938 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:32:27.991291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991124 5938 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:32:27.991353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991293 5938 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:32:27.991353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991309 5938 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:32:27.991353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991335 5938 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:32:27.991353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991345 5938 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:32:27.991353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991353 5938 policy_none.go:50] "Start" Oct 30 00:32:27.991561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991386 5938 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:32:27.991561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.991401 5938 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:32:27.992054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.992034 5938 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:32:27.992054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.992052 5938 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:32:27.992190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.992060 5938 policy_none.go:44] "Start" Oct 30 00:32:27.997202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:27.997170 5938 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:32:28.001926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.001906 5938 manager.go:341] "Starting Device Plugin manager" Oct 30 00:32:28.002022 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.001941 5938 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:32:28.002022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.001953 5938 server.go:87] "Starting device plugin registration server" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.003539 5938 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.003559 5938 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.003986 5938 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.004120 5938 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.004167 5938 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.004174 5938 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:32:28.009127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.004255 5938 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:32:28.004151573Z","UUID":"608964fa-8c35-4e78-b6de-6f97595fab55","Handler":null,"Name":"","Endpoint":""} Oct 30 00:32:28.009824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.009713 5938 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:32:28.009824 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.009728 5938 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:32:28.009824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.009743 5938 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:32:28.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.103675 5938 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:32:28.105705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.105688 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:32:28.105770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.105719 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:32:28.105770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.105729 5938 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-130.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:32:28.105852 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.105794 5938 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.139486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.139462 5938 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.139581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.139522 5938 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.168747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.168704 5938 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal"] Oct 30 00:32:28.169336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169308 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerStarted","Data":"b7ca537036736941b52cbddc1be48522cded10595af2bb44f951cd491a4a6551"} Oct 30 00:32:28.169414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169343 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerDied","Data":"137309faba7fea05098e6d9415726070ef5c0692681dead6b25c0fc1ddf2e974"} Oct 30 00:32:28.169414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169358 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" event={"ID":"5b21b9005987db40e0d6504b950f5deb","Type":"ContainerStarted","Data":"488f10734b57ff9d319f53fd0940073ecba6a8e3c93a14befe4700e49a09ca2a"} Oct 30 00:32:28.169414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169369 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" event={"ID":"87b9cc45a336a6ed23d0ff1b116a4584","Type":"ContainerStarted","Data":"f5383eb38ea54bfb44a337414777ba91ec54268919d4cf77259ca2256a034e47"} Oct 30 00:32:28.169414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169380 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" event={"ID":"87b9cc45a336a6ed23d0ff1b116a4584","Type":"ContainerStarted","Data":"ce3369cfc7436f5b494b2617665974034c00ff2b17d02e4fb61311022dd9bc71"} Oct 30 00:32:28.169548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169514 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.169608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.169595 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.192359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.192337 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:32:28.192446 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.192372 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.193023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.193006 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:32:28.193061 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.193040 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.287439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.287363 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.287439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.287395 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.287439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.287415 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388374 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388395 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388440 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388458 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5b21b9005987db40e0d6504b950f5deb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" (UID: \"5b21b9005987db40e0d6504b950f5deb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.388529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.388440 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b9cc45a336a6ed23d0ff1b116a4584-config\") pod \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" (UID: \"87b9cc45a336a6ed23d0ff1b116a4584\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.825196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.825166 5938 apiserver.go:52] "Watching apiserver" Oct 30 00:32:28.866781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.866750 5938 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:32:28.867514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.867490 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzdmx","openshift-ovn-kubernetes/ovnkube-node-8mdqj","openshift-monitoring/alertmanager-main-0","openshift-dns/dns-default-846tj","openshift-multus/multus-qmmbh","openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v","openshift-insights/insights-runtime-extractor-mcpcw","openshift-monitoring/metrics-server-56f7b49654-f62f5","openshift-multus/network-metrics-daemon-n4f84","openshift-multus/multus-additional-cni-plugins-pbljz","openshift-network-operator/iptables-alerter-cl8gl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm","openshift-image-registry/node-ca-5f7mc","openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal","openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl","kube-system/global-pull-secret-syncer-h282r","openshift-ingress-canary/ingress-canary-gptfr","kube-system/konnectivity-agent-5zk8x","kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-f4q86","openshift-cluster-node-tuning-operator/tuned-nwh2w","openshift-console/console-546cc56848-dbr26","openshift-console/console-6ccc9dd444-ng59m","openshift-dns/node-resolver-m75x5"] Oct 30 00:32:28.883175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.883140 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.883544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.883508 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:32:28.883645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.883582 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.884251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884220 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:32:28.884361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884239 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.884418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884360 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:32:28.884508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884487 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\"" Oct 30 00:32:28.884566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884521 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\"" Oct 30 00:32:28.884705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884677 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:32:28.884762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884753 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\"" Oct 30 00:32:28.884846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884833 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:32:28.884925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884907 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.884983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.884970 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:32:28.885046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.885030 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\"" Oct 30 00:32:28.894818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.894795 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.895476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.895461 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.895942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.895924 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:32:28.895987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.895948 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.895987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.895924 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.895987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.895977 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896078 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896111 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896118 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:32:28.896141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896133 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896154 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896164 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896156 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896194 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896197 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:32:28.896261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896236 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896195 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896304 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896319 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896416 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896421 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896443 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896446 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.896483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896456 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 00:32:28.896711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896496 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:32:28.896711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896534 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:32:28.896711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.896558 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:32:28.897817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.897799 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:32:28.898121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898101 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\"" Oct 30 00:32:28.898190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898122 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\"" Oct 30 00:32:28.898190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898156 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\"" Oct 30 00:32:28.898190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898162 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\"" Oct 30 00:32:28.898190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898176 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:32:28.898190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898164 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:32:28.898374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898197 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 00:32:28.898374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898293 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\"" Oct 30 00:32:28.898374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898314 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\"" Oct 30 00:32:28.898374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898352 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:32:28.898622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.898604 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\"" Oct 30 00:32:28.940897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.940858 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.941096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.941079 5938 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.942395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.942378 5938 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:32:28.963408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.963390 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:32:28.963473 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.963424 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.966701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.966687 5938 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:32:28.966738 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:28.966723 5938 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-130.ec2.internal" Oct 30 00:32:28.985777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.985760 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.986027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986013 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.986293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986275 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:32:28.986371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986274 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:32:28.986371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986293 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:32:28.986519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986499 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:32:28.986582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986565 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:32:28.986632 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986615 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:32:28.986701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986676 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:32:28.986701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986688 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:32:28.986880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986705 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 00:32:28.986880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986717 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 00:32:28.986880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986723 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 00:32:28.986880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986778 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 00:32:28.986880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986815 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 00:32:28.987081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986956 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 00:32:28.987081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.986969 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:32:28.987081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987005 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:32:28.987200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987109 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:32:28.987200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987132 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 00:32:28.987200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987152 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:32:28.987200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987191 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:32:28.987378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987266 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:32:28.987435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987363 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:32:28.987435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987372 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:32:28.987435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987380 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987486 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-skp7j\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987512 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987519 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987519 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987532 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 00:32:28.987578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987490 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987619 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987633 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987645 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w25sk\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987634 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-78xdz\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987721 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qncjw\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987744 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987764 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:32:28.987808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987765 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987850 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5vxhs\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987875 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987895 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.987928 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988014 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:32:28.988098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988065 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:32:28.988292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988109 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ju4vrtprvtbp\"" Oct 30 00:32:28.988292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988172 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kz97h\"" Oct 30 00:32:28.988292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988279 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q4g\"" Oct 30 00:32:28.988398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988311 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8pprt\"" Oct 30 00:32:28.988398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988364 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 00:32:28.988511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988413 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 00:32:28.988511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988430 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 00:32:28.988618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988606 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x662k\"" Oct 30 00:32:28.988737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.988723 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:32:28.991451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991428 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.991507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktzs\" (UniqueName: \"kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.991507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991476 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:28.991507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.991636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991524 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.991636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991549 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.991636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3757f105-2260-48d2-be57-41ba8481ae63-audit-log\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.991813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991770 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.991813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991809 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.991920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991803 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.991920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991885 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.992031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991933 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-textfile\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.992031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991952 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:28.992031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.991989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:28.992031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992064 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlsr\" (UniqueName: \"kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992077 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-client-certs\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992108 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79858a62-0cea-444e-a2ce-fec46d542e55-data-volume\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.992170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992160 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992197 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992209 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992229 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992263 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4859\" (UniqueName: \"kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992308 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992309 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992338 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992426 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992443 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992469 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73ed2348-6522-439c-9381-92240fabb46d-iptables-alerter-script\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.992498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992483 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992584 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992628 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992653 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992676 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992703 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992738 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992830 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.992850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992825 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992892 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992901 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992922 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992959 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992979 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.992999 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993028 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993032 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4ed79ad-9a13-4899-8307-eff21d88d3e6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993044 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993059 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9rr\" (UniqueName: \"kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993076 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993092 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.993120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993116 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993133 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993135 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993180 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-tmp-dir\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993186 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993218 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993239 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993276 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993293 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993308 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993343 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-env-overrides\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993367 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbbs\" (UniqueName: \"kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993467 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993588 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993594 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:28.993652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993652 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993696 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993695 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-tls\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993699 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993744 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993781 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9e691205-7020-4ddd-9fe6-61bb9d460d30-original-pull-secret\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993820 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-metrics-client-ca\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993935 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993956 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-config\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993976 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.993960 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/56772ce8-a261-4951-b032-2b02e0f486ec-serviceca\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994045 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994051 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994065 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994064 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994083 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994365 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994415 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.994504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994531 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994594 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5jrm\" (UniqueName: \"kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994625 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994684 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995076 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79858a62-0cea-444e-a2ce-fec46d542e55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995083 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.994689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjzq\" (UniqueName: \"kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995138 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995414 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-secret-metrics-server-tls\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995484 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995530 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995565 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995591 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.995642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995635 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995668 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfd7\" (UniqueName: \"kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995737 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995794 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995827 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995847 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995944 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.995983 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996020 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996057 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996093 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08604272-a8a8-44c9-bcc1-9b3f12a37d77-config-volume\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996161 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996197 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bdb\" (UniqueName: \"kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996238 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996270 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbjc\" (UniqueName: \"kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996305 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996341 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.996379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996388 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:28.997349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.996508 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.997349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997043 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-cert\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:28.997349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997344 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.997608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997588 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovn-node-metrics-cert\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.997679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997651 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\"" Oct 30 00:32:28.997763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997745 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08604272-a8a8-44c9-bcc1-9b3f12a37d77-tmp-dir\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:28.997823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.997903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.997827 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:28.998022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998004 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gsf\" (UniqueName: \"kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.998149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998130 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.998382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998362 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.998460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:28.998460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998416 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.998565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998467 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1huu4egidotfk\"" Oct 30 00:32:28.998565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998477 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:28.998565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998515 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 00:32:28.998565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.998754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.998754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998593 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.998754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998638 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9pt\" (UniqueName: \"kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:28.998754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.998983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.998938 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.999237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999211 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:28.999330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:28.999830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999808 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkgx\" (UniqueName: \"kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:28.999928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999851 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:28.999928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999900 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw2w\" (UniqueName: \"kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:29.000060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqc5\" (UniqueName: \"kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:29.000060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.000060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:28.999975 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000109 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000147 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000160 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000182 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000230 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnslc\" (UniqueName: \"kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:29.000270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000300 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000369 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000414 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhj5c\" (UniqueName: \"kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000488 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000537 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000557 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000654 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.000694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000693 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000719 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000749 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000773 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000800 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000806 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000822 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000833 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-metrics-server-audit-profiles\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000847 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000891 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000897 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000929 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.000956 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001004 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001032 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001058 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001079 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001101 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001157 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.001232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001249 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-konnectivity-ca\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001255 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001268 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001285 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001257 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001378 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001440 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001478 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc-agent-certs\") pod \"konnectivity-agent-5zk8x\" (UID: \"1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc\") " pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001514 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001594 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001585 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001630 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001648 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001662 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001782 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001806 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001808 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001823 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001839 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-metrics-certs\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001849 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.001968 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002020 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-tmp\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002052 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002053 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-tuned\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.002098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002086 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002123 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002151 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tzgc\" (UniqueName: \"kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002201 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002202 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002226 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002242 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002275 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002289 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002325 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002368 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002374 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002421 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002441 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002500 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002526 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002579 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08604272-a8a8-44c9-bcc1-9b3f12a37d77-metrics-tls\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002587 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002620 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-cni-binary-copy\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002724 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002741 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79358397-9466-4f7b-9349-cd976b869ead-multus-daemon-config\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002755 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002782 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3757f105-2260-48d2-be57-41ba8481ae63-client-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002832 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7876ee6c-178d-4ca6-923a-d76c0867bd85-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002838 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.002970 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79858a62-0cea-444e-a2ce-fec46d542e55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.003065 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34e4db94-de44-4a89-9bfe-5ac1d5997038-ovnkube-script-lib\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.003357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.003089 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3757f105-2260-48d2-be57-41ba8481ae63-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.004533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.003729 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.007485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.007466 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:29.008721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.008704 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.012314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.012294 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.013551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.013533 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktzs\" (UniqueName: \"kubernetes.io/projected/c2b8acd5-22da-46b5-a40d-7e45cafcc342-kube-api-access-qktzs\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.013641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.013626 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9tb\" (UniqueName: \"kubernetes.io/projected/42079552-0ddd-4e83-85ef-604e435466eb-kube-api-access-kp9tb\") pod \"network-check-target-f4q86\" (UID: \"42079552-0ddd-4e83-85ef-604e435466eb\") " pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:29.013690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.013668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"console-6ccc9dd444-ng59m\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:29.013771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.013757 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlsr\" (UniqueName: \"kubernetes.io/projected/08604272-a8a8-44c9-bcc1-9b3f12a37d77-kube-api-access-lmlsr\") pod \"dns-default-846tj\" (UID: \"08604272-a8a8-44c9-bcc1-9b3f12a37d77\") " pod="openshift-dns/dns-default-846tj" Oct 30 00:32:29.013978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.013963 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"alertmanager-main-0\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:29.015399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.015385 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4859\" (UniqueName: \"kubernetes.io/projected/84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc-kube-api-access-x4859\") pod \"openshift-state-metrics-7bd7489957-4qvj6\" (UID: \"84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-4qvj6" Oct 30 00:32:29.017544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.017525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjzq\" (UniqueName: \"kubernetes.io/projected/79358397-9466-4f7b-9349-cd976b869ead-kube-api-access-bgjzq\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.019317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.019302 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9rr\" (UniqueName: \"kubernetes.io/projected/34e4db94-de44-4a89-9bfe-5ac1d5997038-kube-api-access-ls9rr\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.019911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.019896 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbbs\" (UniqueName: \"kubernetes.io/projected/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-kube-api-access-nrbbs\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:29.020217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.020203 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"console-546cc56848-dbr26\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:29.024390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.024363 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5jrm\" (UniqueName: \"kubernetes.io/projected/3757f105-2260-48d2-be57-41ba8481ae63-kube-api-access-k5jrm\") pod \"metrics-server-56f7b49654-f62f5\" (UID: \"3757f105-2260-48d2-be57-41ba8481ae63\") " pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:29.024550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.024531 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfd7\" (UniqueName: \"kubernetes.io/projected/c9149ff3-efaf-4600-b5fe-7699ffcd8db5-kube-api-access-6qfd7\") pod \"prometheus-operator-9cd89b74c-r5ptl\" (UID: \"c9149ff3-efaf-4600-b5fe-7699ffcd8db5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r5ptl" Oct 30 00:32:29.024765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.024745 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbjc\" (UniqueName: \"kubernetes.io/projected/79858a62-0cea-444e-a2ce-fec46d542e55-kube-api-access-dpbjc\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:29.024830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.024808 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gsf\" (UniqueName: \"kubernetes.io/projected/d4ed79ad-9a13-4899-8307-eff21d88d3e6-kube-api-access-n2gsf\") pod \"kube-state-metrics-5dc8d7d89b-2jd9q\" (UID: \"d4ed79ad-9a13-4899-8307-eff21d88d3e6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2jd9q" Oct 30 00:32:29.024905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.024832 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"prometheus-k8s-0\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:29.027235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.027219 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkgx\" (UniqueName: \"kubernetes.io/projected/6283ace2-25a6-43fe-b12c-1d952c5c95da-kube-api-access-ztkgx\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.027816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.027800 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw2w\" (UniqueName: \"kubernetes.io/projected/7876ee6c-178d-4ca6-923a-d76c0867bd85-kube-api-access-rbw2w\") pod \"monitoring-plugin-77f4f76594-fqf4v\" (UID: \"7876ee6c-178d-4ca6-923a-d76c0867bd85\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:29.029362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.029343 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnslc\" (UniqueName: \"kubernetes.io/projected/661abd2b-095c-4b75-ae58-a5c2a82ad0ff-kube-api-access-tnslc\") pod \"network-metrics-daemon-n4f84\" (UID: \"661abd2b-095c-4b75-ae58-a5c2a82ad0ff\") " pod="openshift-multus/network-metrics-daemon-n4f84" Oct 30 00:32:29.029730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.029712 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqc5\" (UniqueName: \"kubernetes.io/projected/56772ce8-a261-4951-b032-2b02e0f486ec-kube-api-access-4pqc5\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:29.029931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.029914 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9pt\" (UniqueName: \"kubernetes.io/projected/c7003a5d-b3e4-4c2e-88d2-f362a5f78180-kube-api-access-bz9pt\") pod \"ingress-canary-gptfr\" (UID: \"c7003a5d-b3e4-4c2e-88d2-f362a5f78180\") " pod="openshift-ingress-canary/ingress-canary-gptfr" Oct 30 00:32:29.030024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.030010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tzgc\" (UniqueName: \"kubernetes.io/projected/73ed2348-6522-439c-9381-92240fabb46d-kube-api-access-5tzgc\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:29.030577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.030563 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhj5c\" (UniqueName: \"kubernetes.io/projected/24461adc-8328-4dd9-a4ac-859666bdc68c-kube-api-access-bhj5c\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.031107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.031091 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bdb\" (UniqueName: \"kubernetes.io/projected/6bf30350-e3f7-4edd-a691-b499f83bccb6-kube-api-access-t9bdb\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103337 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.103448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103367 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103393 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:29.103448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103418 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:29.103448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103434 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-multus\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.103448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103443 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103467 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73ed2348-6522-439c-9381-92240fabb46d-host-slash\") pod \"iptables-alerter-cl8gl\" (UID: \"73ed2348-6522-439c-9381-92240fabb46d\") " pod="openshift-network-operator/iptables-alerter-cl8gl" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103479 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-node-log\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103466 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103491 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34aaeeff-dda3-47f1-84cf-ea29bb1bd772-hosts-file\") pod \"node-resolver-m75x5\" (UID: \"34aaeeff-dda3-47f1-84cf-ea29bb1bd772\") " pod="openshift-dns/node-resolver-m75x5" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103523 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-modprobe-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103549 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79858a62-0cea-444e-a2ce-fec46d542e55-crio-socket\") pod \"insights-runtime-extractor-mcpcw\" (UID: \"79858a62-0cea-444e-a2ce-fec46d542e55\") " pod="openshift-insights/insights-runtime-extractor-mcpcw" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103561 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103564 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-lib-modules\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103576 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-bin\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103613 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103642 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103665 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103696 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103703 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-run\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.103707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103710 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-sys-fs\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103721 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103741 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-kubelet-config\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103744 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103778 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-multus-certs\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103778 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-cni-netd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103793 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103827 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103835 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-device-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103847 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103905 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-sys\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103913 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-k8s-cni-cncf-io\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103917 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103936 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-run-netns\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103945 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103956 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-registration-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103961 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103993 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysconfig\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103993 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.103971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-kubernetes\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104023 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104032 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-var-lib-kubelet\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104052 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-etc-kubernetes\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104069 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104089 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104105 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104119 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104140 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-cnibin\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104143 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104175 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104179 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-root\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104191 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104194 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104215 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-slash\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104222 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-d\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-hostroot\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104237 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104257 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-systemd\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104263 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-log-socket\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104268 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104291 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104268 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-host\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104306 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104321 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104323 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-ovn-kubernetes\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104333 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2b8acd5-22da-46b5-a40d-7e45cafcc342-system-cni-dir\") pod \"multus-additional-cni-plugins-pbljz\" (UID: \"c2b8acd5-22da-46b5-a40d-7e45cafcc342\") " pod="openshift-multus/multus-additional-cni-plugins-pbljz" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104341 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-systemd\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104365 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104375 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104384 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-socket-dir-parent\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104392 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104417 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104433 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104447 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104452 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-cni-bin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104463 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104475 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-etc-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104416 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-sys\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104487 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104486 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-run-netns\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.104467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104514 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104518 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104524 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-cnibin\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104570 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9e691205-7020-4ddd-9fe6-61bb9d460d30-dbus\") pod \"global-pull-secret-syncer-h282r\" (UID: \"9e691205-7020-4ddd-9fe6-61bb9d460d30\") " pod="kube-system/global-pull-secret-syncer-h282r" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104597 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-var-lib-openvswitch\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104602 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104622 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6283ace2-25a6-43fe-b12c-1d952c5c95da-node-exporter-wtmp\") pod \"node-exporter-mzdmx\" (UID: \"6283ace2-25a6-43fe-b12c-1d952c5c95da\") " pod="openshift-monitoring/node-exporter-mzdmx" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104650 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104655 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-systemd-units\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6bf30350-e3f7-4edd-a691-b499f83bccb6-etc-sysctl-conf\") pod \"tuned-nwh2w\" (UID: \"6bf30350-e3f7-4edd-a691-b499f83bccb6\") " pod="openshift-cluster-node-tuning-operator/tuned-nwh2w" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104682 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56772ce8-a261-4951-b032-2b02e0f486ec-host\") pod \"node-ca-5f7mc\" (UID: \"56772ce8-a261-4951-b032-2b02e0f486ec\") " pod="openshift-image-registry/node-ca-5f7mc" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104722 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/24461adc-8328-4dd9-a4ac-859666bdc68c-socket-dir\") pod \"aws-ebs-csi-driver-node-wctdm\" (UID: \"24461adc-8328-4dd9-a4ac-859666bdc68c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wctdm" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104735 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104757 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-system-cni-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104765 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104791 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104801 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-host-var-lib-kubelet\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104822 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104827 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-host-kubelet\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104847 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79358397-9466-4f7b-9349-cd976b869ead-multus-conf-dir\") pod \"multus-qmmbh\" (UID: \"79358397-9466-4f7b-9349-cd976b869ead\") " pod="openshift-multus/multus-qmmbh" Oct 30 00:32:29.106396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:29.104921 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34e4db94-de44-4a89-9bfe-5ac1d5997038-run-ovn\") pod \"ovnkube-node-8mdqj\" (UID: \"34e4db94-de44-4a89-9bfe-5ac1d5997038\") " pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:30.674195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:30.674155 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:30.690393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:30.690361 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:31.384019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:31.383980 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:32:31.597948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:31.597920 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:31.646148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:31.646079 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-846tj" Oct 30 00:32:32.492070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:32.492039 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:32.746448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:32.746361 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fqf4v" Oct 30 00:32:33.848079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:33.848039 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:33.925379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:33.925353 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:33.925557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:33.925544 5938 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:32:34.082484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:34.082452 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:34.298345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:34.298305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:34.440013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:34.439983 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:37.533322 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:37.533285 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:38.750765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:38.750730 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8mdqj" Oct 30 00:32:39.575551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:39.575512 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5zk8x" Oct 30 00:32:41.152286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:41.152251 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56f7b49654-f62f5" Oct 30 00:32:41.603373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:41.603345 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:41.607307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:41.607282 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:32:42.045033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:42.044957 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:42.205653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:42.205620 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.415323 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:43.415290 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.572345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:43.572315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:43.722156 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:43.722076 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:45.769110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:45.769078 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:45.938973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:45.938942 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:46.099820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:46.099750 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:47.209778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.209741 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:47.210277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210056 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="alertmanager" containerID="cri-o://39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905" gracePeriod=120 Oct 30 00:32:47.210277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210125 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-metric" containerID="cri-o://c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c" gracePeriod=120 Oct 30 00:32:47.210277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210135 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-web" containerID="cri-o://be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d" gracePeriod=120 Oct 30 00:32:47.210398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210152 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="config-reloader" containerID="cri-o://9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e" gracePeriod=120 Oct 30 00:32:47.211144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210186 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy" containerID="cri-o://b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1" gracePeriod=120 Oct 30 00:32:47.211144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:47.210739 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="prom-label-proxy" containerID="cri-o://2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b" gracePeriod=120 Oct 30 00:32:48.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014174 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b" exitCode=0 Oct 30 00:32:48.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014203 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c" exitCode=0 Oct 30 00:32:48.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014211 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1" exitCode=0 Oct 30 00:32:48.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014220 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e" exitCode=0 Oct 30 00:32:48.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014228 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905" exitCode=0 Oct 30 00:32:48.014533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014256 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b"} Oct 30 00:32:48.014533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014328 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c"} Oct 30 00:32:48.014533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014372 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1"} Oct 30 00:32:48.014533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e"} Oct 30 00:32:48.014533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.014443 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905"} Oct 30 00:32:48.465426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.465404 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:48.649561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.649727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649571 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.649727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649594 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.649727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649627 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.649887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.649887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649830 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649897 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649934 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.649975 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650014 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650043 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650073 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650133 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") pod \"7063ed5c-d12e-4887-812f-4e1eeea42495\" (UID: \"7063ed5c-d12e-4887-812f-4e1eeea42495\") " Oct 30 00:32:48.650912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650808 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:48.651036 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.650941 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:48.651036 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.651024 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:48.653027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.652968 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.653217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.653146 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:48.653397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.653256 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.653651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.653624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "kube-api-access-fb89b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:48.653827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.653794 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.654142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.654110 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:48.654142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.654117 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.654330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.654295 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.658387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.658354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.664394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.664371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config" pod "7063ed5c-d12e-4887-812f-4e1eeea42495" (UID: "7063ed5c-d12e-4887-812f-4e1eeea42495"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:48.750698 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750671 5938 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750698 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750695 5938 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-config-out\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750732 5938 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-web-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750748 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb89b\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-kube-api-access-fb89b\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750764 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-metrics-client-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750778 5938 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-main-db\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750794 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750808 5938 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7063ed5c-d12e-4887-812f-4e1eeea42495-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750824 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.750846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750839 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.751188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750853 5938 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-cluster-tls-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.751188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750891 5938 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7063ed5c-d12e-4887-812f-4e1eeea42495-secret-alertmanager-main-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:48.751188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:48.750907 5938 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7063ed5c-d12e-4887-812f-4e1eeea42495-tls-assets\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:49.021982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.021954 5938 generic.go:356] "Generic (PLEG): container finished" podID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerID="be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d" exitCode=0 Oct 30 00:32:49.022079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.022008 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d"} Oct 30 00:32:49.022079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.022030 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 00:32:49.022079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.022067 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7063ed5c-d12e-4887-812f-4e1eeea42495","Type":"ContainerDied","Data":"bcbce8fd9737935f4cff97b7a54155c12f5041c58a166254ca16d1588450e67f"} Oct 30 00:32:49.022187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.022102 5938 scope.go:122] "RemoveContainer" containerID="2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b" Oct 30 00:32:49.030110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.030092 5938 scope.go:122] "RemoveContainer" containerID="c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c" Oct 30 00:32:49.036923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.036907 5938 scope.go:122] "RemoveContainer" containerID="b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1" Oct 30 00:32:49.045965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.044425 5938 scope.go:122] "RemoveContainer" containerID="be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d" Oct 30 00:32:49.052704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.052689 5938 scope.go:122] "RemoveContainer" containerID="9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e" Oct 30 00:32:49.059381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.059347 5938 scope.go:122] "RemoveContainer" containerID="39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905" Oct 30 00:32:49.066776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.066762 5938 scope.go:122] "RemoveContainer" containerID="57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039" Oct 30 00:32:49.073535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.073521 5938 scope.go:122] "RemoveContainer" containerID="2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b" Oct 30 00:32:49.073824 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.073800 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b\": container with ID starting with 2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b not found: ID does not exist" containerID="2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b" Oct 30 00:32:49.074019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.073832 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b"} err="failed to get container status \"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b\": rpc error: code = NotFound desc = could not find container \"2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b\": container with ID starting with 2adba9190eb3aa63b760af363ba6a3094ae75eac69448c75cb2206bba142455b not found: ID does not exist" Oct 30 00:32:49.074019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.073872 5938 scope.go:122] "RemoveContainer" containerID="c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c" Oct 30 00:32:49.074108 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.074092 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c\": container with ID starting with c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c not found: ID does not exist" containerID="c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c" Oct 30 00:32:49.074142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074112 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c"} err="failed to get container status \"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c\": rpc error: code = NotFound desc = could not find container \"c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c\": container with ID starting with c1b5749e806f89357a2d119ac062231b3e8222500c45024c687e0888c35d2e6c not found: ID does not exist" Oct 30 00:32:49.074142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074124 5938 scope.go:122] "RemoveContainer" containerID="b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1" Oct 30 00:32:49.074362 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.074342 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1\": container with ID starting with b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1 not found: ID does not exist" containerID="b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1" Oct 30 00:32:49.074401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074369 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1"} err="failed to get container status \"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1\": rpc error: code = NotFound desc = could not find container \"b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1\": container with ID starting with b68ff254a12d3b3f1b7b869905d8d2ca08aa47cfb446a54f2503a573bb7650d1 not found: ID does not exist" Oct 30 00:32:49.074401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074385 5938 scope.go:122] "RemoveContainer" containerID="be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d" Oct 30 00:32:49.074615 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.074597 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d\": container with ID starting with be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d not found: ID does not exist" containerID="be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d" Oct 30 00:32:49.074665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074619 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d"} err="failed to get container status \"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d\": rpc error: code = NotFound desc = could not find container \"be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d\": container with ID starting with be912bd0e96111d82d8b3631a943e8cb1e858b0ff7a9ff1eb57fb1a199a2d83d not found: ID does not exist" Oct 30 00:32:49.074665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074632 5938 scope.go:122] "RemoveContainer" containerID="9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e" Oct 30 00:32:49.074847 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.074832 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e\": container with ID starting with 9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e not found: ID does not exist" containerID="9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e" Oct 30 00:32:49.074903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074849 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e"} err="failed to get container status \"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e\": rpc error: code = NotFound desc = could not find container \"9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e\": container with ID starting with 9c72ffd804ab37d4970989e3c665310eba30f03b3370a62c14b6ef34a3656f0e not found: ID does not exist" Oct 30 00:32:49.074903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.074873 5938 scope.go:122] "RemoveContainer" containerID="39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905" Oct 30 00:32:49.075128 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.075114 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905\": container with ID starting with 39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905 not found: ID does not exist" containerID="39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905" Oct 30 00:32:49.075170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.075134 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905"} err="failed to get container status \"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905\": rpc error: code = NotFound desc = could not find container \"39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905\": container with ID starting with 39f59cf119e04f1f49f9ed6c1d54506025b3e9ef94e1c85650eb673322a24905 not found: ID does not exist" Oct 30 00:32:49.075170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.075146 5938 scope.go:122] "RemoveContainer" containerID="57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039" Oct 30 00:32:49.075346 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:49.075330 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039\": container with ID starting with 57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039 not found: ID does not exist" containerID="57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039" Oct 30 00:32:49.075384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.075350 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039"} err="failed to get container status \"57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039\": rpc error: code = NotFound desc = could not find container \"57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039\": container with ID starting with 57fa2dfb196cbe8a01dfeac93199a477d930a44906ea0877cb73c903f6f6b039 not found: ID does not exist" Oct 30 00:32:49.088831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.088809 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:49.101308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.101286 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 00:32:49.305402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.305317 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:49.471811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.471773 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:49.687974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.687948 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:49.873779 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:49.873739 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" path="/var/lib/kubelet/pods/7063ed5c-d12e-4887-812f-4e1eeea42495/volumes" Oct 30 00:32:51.875977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.875944 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:51.876997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.876939 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="prometheus" containerID="cri-o://b8d357332c37156f88eec1234009fb5c9d943d313fd513c2b1093982bff96e90" gracePeriod=600 Oct 30 00:32:51.877176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.877149 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-web" containerID="cri-o://91d348f596e61e2651462a869681353be154d365933d0e2c8762c2aeaaa5f42b" gracePeriod=600 Oct 30 00:32:51.877293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.877165 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8cc5d49e2b1d6cb95d6f24a6335f407c5a76e1490cf548eab0e930614c699b2e" gracePeriod=600 Oct 30 00:32:51.877293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.877180 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="config-reloader" containerID="cri-o://880d9d2fda5b9d1a6f1ebb9327ab0c092aa91827439a24ee70bc95569e9ea30a" gracePeriod=600 Oct 30 00:32:51.877293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.876961 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy" containerID="cri-o://125daffba8f0319d9001ff92d63841ff6f772d9519d761d6d1702713e1155818" gracePeriod=600 Oct 30 00:32:51.877293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:51.877123 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="thanos-sidecar" containerID="cri-o://bccaedccc1d4dda885b59f87d1aa89987ee160780cadd0e0419558d2fbcef8a1" gracePeriod=600 Oct 30 00:32:52.039501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039471 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="8cc5d49e2b1d6cb95d6f24a6335f407c5a76e1490cf548eab0e930614c699b2e" exitCode=0 Oct 30 00:32:52.039501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039498 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="125daffba8f0319d9001ff92d63841ff6f772d9519d761d6d1702713e1155818" exitCode=0 Oct 30 00:32:52.039501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039507 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="91d348f596e61e2651462a869681353be154d365933d0e2c8762c2aeaaa5f42b" exitCode=0 Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039513 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="bccaedccc1d4dda885b59f87d1aa89987ee160780cadd0e0419558d2fbcef8a1" exitCode=0 Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039519 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="880d9d2fda5b9d1a6f1ebb9327ab0c092aa91827439a24ee70bc95569e9ea30a" exitCode=0 Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039525 5938 generic.go:356] "Generic (PLEG): container finished" podID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerID="b8d357332c37156f88eec1234009fb5c9d943d313fd513c2b1093982bff96e90" exitCode=0 Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039545 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"8cc5d49e2b1d6cb95d6f24a6335f407c5a76e1490cf548eab0e930614c699b2e"} Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039634 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"125daffba8f0319d9001ff92d63841ff6f772d9519d761d6d1702713e1155818"} Oct 30 00:32:52.039689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039680 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"91d348f596e61e2651462a869681353be154d365933d0e2c8762c2aeaaa5f42b"} Oct 30 00:32:52.039884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039719 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"bccaedccc1d4dda885b59f87d1aa89987ee160780cadd0e0419558d2fbcef8a1"} Oct 30 00:32:52.039884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039768 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"880d9d2fda5b9d1a6f1ebb9327ab0c092aa91827439a24ee70bc95569e9ea30a"} Oct 30 00:32:52.039884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.039807 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"b8d357332c37156f88eec1234009fb5c9d943d313fd513c2b1093982bff96e90"} Oct 30 00:32:52.130428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.130379 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:52.277170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277136 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277189 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277224 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277264 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277295 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277361 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277441 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277479 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277509 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277518 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277558 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277619 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277648 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.277681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277653 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277729 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277766 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277762 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.277806 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.278153 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:52.278333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.278164 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.278632 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.278449 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279131 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279232 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") pod \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\" (UID: \"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694\") " Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279510 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279532 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279551 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279565 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-metrics-client-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279580 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-db\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.279876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.279594 5938 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.281972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.281920 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.282436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.282407 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:52.282783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.282754 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:52.283492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.283464 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.283587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.283542 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.283649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.283605 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "kube-api-access-72gs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:52.283708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.283687 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.283970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.283944 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.284339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.284306 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.284475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.284453 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:52.284475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.284463 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.285645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.285618 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.285725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.285667 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.293115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.293092 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config" pod "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" (UID: "06a9180c-aa5e-4d6b-9c47-25ffcd5fb694"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:52.380753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380693 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.380753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380722 5938 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config-out\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.380753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380736 5938 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-tls-assets\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.380753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380750 5938 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380788 5938 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-web-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380802 5938 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380815 5938 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380829 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gs7\" (UniqueName: \"kubernetes.io/projected/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-kube-api-access-72gs7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380841 5938 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-grpc-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380854 5938 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-kube-rbac-proxy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380888 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380903 5938 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-metrics-client-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380916 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:52.381023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:52.380931 5938 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:53.052153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.052120 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a9180c-aa5e-4d6b-9c47-25ffcd5fb694","Type":"ContainerDied","Data":"e35688a19a204bedae8fdeab3162e4e319d7fe5b161ee7e99aebd7d81c8f801f"} Oct 30 00:32:53.052623 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.052195 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.052623 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.052247 5938 scope.go:122] "RemoveContainer" containerID="8cc5d49e2b1d6cb95d6f24a6335f407c5a76e1490cf548eab0e930614c699b2e" Oct 30 00:32:53.060532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.060509 5938 scope.go:122] "RemoveContainer" containerID="125daffba8f0319d9001ff92d63841ff6f772d9519d761d6d1702713e1155818" Oct 30 00:32:53.068069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.068047 5938 scope.go:122] "RemoveContainer" containerID="91d348f596e61e2651462a869681353be154d365933d0e2c8762c2aeaaa5f42b" Oct 30 00:32:53.075176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.075159 5938 scope.go:122] "RemoveContainer" containerID="bccaedccc1d4dda885b59f87d1aa89987ee160780cadd0e0419558d2fbcef8a1" Oct 30 00:32:53.081902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.081877 5938 scope.go:122] "RemoveContainer" containerID="880d9d2fda5b9d1a6f1ebb9327ab0c092aa91827439a24ee70bc95569e9ea30a" Oct 30 00:32:53.088558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.088536 5938 scope.go:122] "RemoveContainer" containerID="b8d357332c37156f88eec1234009fb5c9d943d313fd513c2b1093982bff96e90" Oct 30 00:32:53.095646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.095628 5938 scope.go:122] "RemoveContainer" containerID="72e98b4e42c17edf9f57b20b784ea8c84a222392e3cad7636cc6ee0b10458141" Oct 30 00:32:53.117589 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.117566 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:53.129434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.129407 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:53.164099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.163766 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:53.164099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164059 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-metric" Oct 30 00:32:53.164099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164072 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-metric" Oct 30 00:32:53.164099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164084 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy" Oct 30 00:32:53.164099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164090 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164097 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="prometheus" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164116 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="prometheus" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164130 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="thanos-sidecar" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164135 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="thanos-sidecar" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164145 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="init-config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164149 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="init-config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164157 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-thanos" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164161 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-thanos" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164169 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164173 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164183 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164187 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164193 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="alertmanager" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164199 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="alertmanager" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164211 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164216 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164223 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164227 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164234 5938 cpu_manager.go:414] "containerMap: removing container" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="init-config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164238 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="init-config-reloader" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164245 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="prom-label-proxy" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164249 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="prom-label-proxy" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164254 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy" Oct 30 00:32:53.164291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164259 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164308 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-metric" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164316 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164322 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="prometheus" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164328 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="thanos-sidecar" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164334 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="init-config-reloader" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164340 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-thanos" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164346 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="config-reloader" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164352 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="config-reloader" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164360 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="alertmanager" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164369 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164374 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="kube-rbac-proxy-web" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164378 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" containerName="init-config-reloader" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164383 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="prom-label-proxy" Oct 30 00:32:53.164964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.164388 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7063ed5c-d12e-4887-812f-4e1eeea42495" containerName="kube-rbac-proxy" Oct 30 00:32:53.167986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.167972 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.186926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.186899 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:32:53.187221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187202 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:32:53.187291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187221 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:32:53.187291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187286 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:32:53.187793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187775 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.187833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187806 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.187833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187825 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.187916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.187898 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188003 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188032 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188047 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188126 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config-out\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188146 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-web-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188172 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmbs\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-kube-api-access-nkmbs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188270 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188286 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188323 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188340 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188394 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188452 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188485 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.188532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.188508 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.191478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191461 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:32:53.191576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191560 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:32:53.191636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191566 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:32:53.191636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191596 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:32:53.191636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191612 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:32:53.191636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.191624 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:32:53.203856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.203832 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kz97h\"" Oct 30 00:32:53.203972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.203908 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:32:53.203972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.203939 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:32:53.203972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.203973 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:32:53.204135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.203845 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ju4vrtprvtbp\"" Oct 30 00:32:53.204135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.204048 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:32:53.204668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.204648 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:53.289043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289016 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289051 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289075 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289171 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289199 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289216 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289236 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config-out\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-web-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289443 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289493 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmbs\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-kube-api-access-nkmbs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289529 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289576 5938 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289622 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.289716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289724 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289799 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289832 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.289844 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.290048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.290734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.290706 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.291300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.291261 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.291468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.291436 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.291683 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.291660 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.292582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.292546 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.292827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.292799 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.293964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.293925 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.293964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.293928 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-web-config\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.294141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.294027 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8233ecf8-dc61-4a5c-a234-bd021ba93fae-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.294141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.294048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.294695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.294671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.294830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.294810 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.294971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.294951 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.295399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.295374 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8233ecf8-dc61-4a5c-a234-bd021ba93fae-config-out\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.295510 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.295485 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.295594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.295576 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8233ecf8-dc61-4a5c-a234-bd021ba93fae-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.312082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.312063 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmbs\" (UniqueName: \"kubernetes.io/projected/8233ecf8-dc61-4a5c-a234-bd021ba93fae-kube-api-access-nkmbs\") pod \"prometheus-k8s-0\" (UID: \"8233ecf8-dc61-4a5c-a234-bd021ba93fae\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.485111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.485056 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:53.770414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.770338 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:53.851269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.851242 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:32:53.854501 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:32:53.854470 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8233ecf8_dc61_4a5c_a234_bd021ba93fae.slice/crio-989d9a12e81c1890c5725560fd86a588159d2a1217ea59687b1670ee77671573 WatchSource:0}: Error finding container 989d9a12e81c1890c5725560fd86a588159d2a1217ea59687b1670ee77671573: Status 404 returned error can't find the container with id 989d9a12e81c1890c5725560fd86a588159d2a1217ea59687b1670ee77671573 Oct 30 00:32:53.874621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.874588 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a9180c-aa5e-4d6b-9c47-25ffcd5fb694" path="/var/lib/kubelet/pods/06a9180c-aa5e-4d6b-9c47-25ffcd5fb694/volumes" Oct 30 00:32:53.895519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.895488 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6ccc9dd444-ng59m" podUID="54406dd0-d002-4fc9-84b4-41532b139761" containerName="console" containerID="cri-o://568215ff2a76e0313bd49e6a1bafb330a77e7ce1f2b160d5912b9682433abeb1" gracePeriod=15 Oct 30 00:32:53.916268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:53.916248 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:32:54.059814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.059782 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"db1966c7942e4dadadd5e00309de671cad6bc0f898126503b29b57d8a279253e"} Oct 30 00:32:54.060207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.059884 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"989d9a12e81c1890c5725560fd86a588159d2a1217ea59687b1670ee77671573"} Oct 30 00:32:54.063542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.063516 5938 generic.go:356] "Generic (PLEG): container finished" podID="54406dd0-d002-4fc9-84b4-41532b139761" containerID="568215ff2a76e0313bd49e6a1bafb330a77e7ce1f2b160d5912b9682433abeb1" exitCode=0 Oct 30 00:32:54.063644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.063554 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc9dd444-ng59m" event={"ID":"54406dd0-d002-4fc9-84b4-41532b139761","Type":"ContainerDied","Data":"568215ff2a76e0313bd49e6a1bafb330a77e7ce1f2b160d5912b9682433abeb1"} Oct 30 00:32:54.187295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.187271 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:54.197523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197491 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197552 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197723 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197792 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197844 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.197923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197902 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.198041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") pod \"54406dd0-d002-4fc9-84b4-41532b139761\" (UID: \"54406dd0-d002-4fc9-84b4-41532b139761\") " Oct 30 00:32:54.198041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.197987 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:54.198139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198087 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:54.198381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198359 5938 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-console-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.198381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198364 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:54.198583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198382 5938 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-trusted-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.198675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:32:54.198750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.198679 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:32:54.200090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.200063 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:54.200287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.200261 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "kube-api-access-bjgqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:32:54.200464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.200428 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert" pod "54406dd0-d002-4fc9-84b4-41532b139761" (UID: "54406dd0-d002-4fc9-84b4-41532b139761"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:32:54.301375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301339 5938 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.301375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301377 5938 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-oauth-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.301596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301427 5938 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54406dd0-d002-4fc9-84b4-41532b139761-oauth-serving-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.301596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301452 5938 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54406dd0-d002-4fc9-84b4-41532b139761-tmp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.301596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301465 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjgqb\" (UniqueName: \"kubernetes.io/projected/54406dd0-d002-4fc9-84b4-41532b139761-kube-api-access-bjgqb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:54.301596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:54.301482 5938 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54406dd0-d002-4fc9-84b4-41532b139761-console-serving-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:32:55.068742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.068715 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccc9dd444-ng59m" Oct 30 00:32:55.068742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.068726 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccc9dd444-ng59m" event={"ID":"54406dd0-d002-4fc9-84b4-41532b139761","Type":"ContainerDied","Data":"45001f8bef284a32073bf321a2768a87f014888a9670251b68c784d1ecc210c8"} Oct 30 00:32:55.069179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.068783 5938 scope.go:122] "RemoveContainer" containerID="568215ff2a76e0313bd49e6a1bafb330a77e7ce1f2b160d5912b9682433abeb1" Oct 30 00:32:55.072897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.072845 5938 generic.go:356] "Generic (PLEG): container finished" podID="8233ecf8-dc61-4a5c-a234-bd021ba93fae" containerID="db1966c7942e4dadadd5e00309de671cad6bc0f898126503b29b57d8a279253e" exitCode=0 Oct 30 00:32:55.073022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.072902 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerDied","Data":"db1966c7942e4dadadd5e00309de671cad6bc0f898126503b29b57d8a279253e"} Oct 30 00:32:55.073022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.072956 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"895e148ac799ab343f569d6b3a0635ef84f709ffb8b02e5a43aa0a08abc70b28"} Oct 30 00:32:55.073022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.072985 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"2929d3d560a64bdb2826fb1dfbd9ab1cdb9ade51be936cdfcf82e1cd286872a0"} Oct 30 00:32:55.073022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.073012 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"ebc201caf449658460dd21493d32b8be813b1a7098c640087bdb34b56618dfea"} Oct 30 00:32:55.073192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.073036 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"6d59215c8eba0ba404fff0c5318d675c0c7161e2ada4d7b7783685ee97c6b08e"} Oct 30 00:32:55.073192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.073063 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"5d2454b223d57cc3328f03c666fa923c539ca20e7c84edb169be1ec62bc95462"} Oct 30 00:32:55.073192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.073094 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8233ecf8-dc61-4a5c-a234-bd021ba93fae","Type":"ContainerStarted","Data":"a27f784b274ee0f620aaa56620dc179c743927180daf0723a6ec22490cd36a04"} Oct 30 00:32:55.118631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.118594 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:55.123018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.123001 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ccc9dd444-ng59m"] Oct 30 00:32:55.188221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.188117 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.162488156 podStartE2EDuration="2.188101811s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.613655ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:55.187347258 +0000 UTC m=+27.427582374" watchObservedRunningTime="2026-10-30 00:32:55.188101811 +0000 UTC m=+27.428336922" Oct 30 00:32:55.874641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:55.874605 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54406dd0-d002-4fc9-84b4-41532b139761" path="/var/lib/kubelet/pods/54406dd0-d002-4fc9-84b4-41532b139761/volumes" Oct 30 00:32:56.152147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:56.152086 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:32:58.485496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:32:58.485460 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:32:58.834584 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:32:58.834506 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:33:01.386500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:01.386461 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f4q86" Oct 30 00:33:04.318686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:04.318641 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:05.829591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:05.829555 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:18.100109 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:18.100072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:21.174484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.174439 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-546cc56848-dbr26" podUID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" containerName="console" containerID="cri-o://bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d" gracePeriod=15 Oct 30 00:33:21.423455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.423431 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:33:21.494572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494511 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494544 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494565 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494587 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494623 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494687 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.494819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494728 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") pod \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\" (UID: \"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc\") " Oct 30 00:33:21.495089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.494813 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:33:21.495089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.495018 5938 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-tmp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.495257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.495213 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:21.495378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.495347 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:21.495378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.495358 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:21.495499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.495387 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:33:21.496807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.496779 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:21.496950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.496845 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "kube-api-access-brxm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:33:21.496950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.496857 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert" pod "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" (UID: "19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:33:21.596339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596313 5938 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596336 5938 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-trusted-ca-bundle\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596347 5938 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-serving-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596356 5938 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596364 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxm8\" (UniqueName: \"kubernetes.io/projected/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-kube-api-access-brxm8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596374 5938 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-console-oauth-config\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:21.596464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:21.596382 5938 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc-oauth-serving-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:33:22.171964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.171894 5938 generic.go:356] "Generic (PLEG): container finished" podID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" containerID="bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d" exitCode=0 Oct 30 00:33:22.171964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.171947 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546cc56848-dbr26" Oct 30 00:33:22.172122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.171983 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546cc56848-dbr26" event={"ID":"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc","Type":"ContainerDied","Data":"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d"} Oct 30 00:33:22.172122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.172047 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546cc56848-dbr26" event={"ID":"19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc","Type":"ContainerDied","Data":"d70f4d71579684f0c4ef2ac83d0866c91f07e39f9131da9766df1b7f410f03f9"} Oct 30 00:33:22.172122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.172070 5938 scope.go:122] "RemoveContainer" containerID="bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d" Oct 30 00:33:22.180927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.180743 5938 scope.go:122] "RemoveContainer" containerID="bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d" Oct 30 00:33:22.181170 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:33:22.181062 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d\": container with ID starting with bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d not found: ID does not exist" containerID="bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d" Oct 30 00:33:22.181170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.181101 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d"} err="failed to get container status \"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d\": rpc error: code = NotFound desc = could not find container \"bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d\": container with ID starting with bce4db326af31d17b29374228e17c473f8c0c1e1fe2fae3833bf67469046de9d not found: ID does not exist" Oct 30 00:33:22.221231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.221202 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:33:22.232593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:22.232570 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546cc56848-dbr26"] Oct 30 00:33:23.116832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:23.116793 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:33:23.133255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:23.133232 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:33:23.190744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:23.190723 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:33:23.873227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:23.873184 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" path="/var/lib/kubelet/pods/19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc/volumes" Oct 30 00:33:28.833522 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:33:28.833481 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:33:33.989834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:33:33.989787 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:33:58.834117 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:33:58.834081 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:34:28.832135 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:34:28.832058 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:34:34.109605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:34:34.109549 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.768881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:34:34.768819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:34.912269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:34:34.912234 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.672102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:34:45.672047 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:45.837042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:34:45.837011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:34:58.835024 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:34:58.834963 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:35:00.710182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:00.710145 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:05.634390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:05.634346 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:05.790135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:05.790072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:07.946427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:07.946395 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:09.309488 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:09.309442 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:11.388598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:11.388556 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:11.529304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:11.529264 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:13.219182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.219154 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:13.439079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.439051 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:13.601791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.601752 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:13.760741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.760709 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62"] Oct 30 00:35:13.761023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761009 5938 cpu_manager.go:414] "containerMap: removing container" podUID="54406dd0-d002-4fc9-84b4-41532b139761" containerName="console" Oct 30 00:35:13.761072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761023 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54406dd0-d002-4fc9-84b4-41532b139761" containerName="console" Oct 30 00:35:13.761072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761046 5938 cpu_manager.go:414] "containerMap: removing container" podUID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" containerName="console" Oct 30 00:35:13.761072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761052 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" containerName="console" Oct 30 00:35:13.761164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761104 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="54406dd0-d002-4fc9-84b4-41532b139761" containerName="console" Oct 30 00:35:13.761164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.761114 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fb59b0-3c5a-4a80-9ddc-c394e1dcd8fc" containerName="console" Oct 30 00:35:13.762974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.762952 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.788802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.788778 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:35:13.788802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.788780 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:35:13.789008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.788814 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zp46w\"" Oct 30 00:35:13.789499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.789479 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62"] Oct 30 00:35:13.857617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.857562 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.857617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.857591 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4p9\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-kube-api-access-6w4p9\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.958677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.958656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.958797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.958682 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4p9\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-kube-api-access-6w4p9\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.984127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.984102 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:13.984240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:13.984221 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4p9\" (UniqueName: \"kubernetes.io/projected/671141fa-f7fe-4f3f-952e-bf6aebb07516-kube-api-access-6w4p9\") pod \"cert-manager-webhook-5dbbbf75cc-xlz62\" (UID: \"671141fa-f7fe-4f3f-952e-bf6aebb07516\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:14.084439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:14.084414 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:14.374427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:14.374397 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62"] Oct 30 00:35:14.377987 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:35:14.377951 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod671141fa_f7fe_4f3f_952e_bf6aebb07516.slice/crio-8513808a860f8f3b80ac0ac1674f8acc1d280a463d17fa376e8e4a7c585c89de WatchSource:0}: Error finding container 8513808a860f8f3b80ac0ac1674f8acc1d280a463d17fa376e8e4a7c585c89de: Status 404 returned error can't find the container with id 8513808a860f8f3b80ac0ac1674f8acc1d280a463d17fa376e8e4a7c585c89de Oct 30 00:35:14.379906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:14.379890 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:35:14.586153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:14.586123 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" event={"ID":"671141fa-f7fe-4f3f-952e-bf6aebb07516","Type":"ContainerStarted","Data":"8513808a860f8f3b80ac0ac1674f8acc1d280a463d17fa376e8e4a7c585c89de"} Oct 30 00:35:16.049472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:16.049436 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:16.359666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:16.359611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:18.371554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:18.371524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:18.518002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:18.517975 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:18.602489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:18.602453 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" event={"ID":"671141fa-f7fe-4f3f-952e-bf6aebb07516","Type":"ContainerStarted","Data":"21fc8aeeee810ccc67c27e6ba68e46df5ad42411ec9cae4e2d4ea74b694d5127"} Oct 30 00:35:18.602651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:18.602583 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:18.641971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:18.641806 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" podStartSLOduration=2.170673244 podStartE2EDuration="5.64173291s" totalImagesPullingTime="3.471059666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:18.641510722 +0000 UTC m=+170.881745835" watchObservedRunningTime="2026-10-30 00:35:18.64173291 +0000 UTC m=+170.881968024" Oct 30 00:35:22.705713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:22.705677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:22.850072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:22.850047 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:23.288419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:23.288385 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:24.607951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:24.607924 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" Oct 30 00:35:28.834003 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:35:28.833967 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:35:30.178494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:30.178461 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:30.326233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:30.326194 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:34.214908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:34.214857 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:35.768653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:35.768617 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:35.934067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:35.934041 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:38.108010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:38.107976 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:47.973029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:47.972946 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:48.176615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.176578 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:48.570576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.570544 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:48.763232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.763204 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk"] Oct 30 00:35:48.765748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.765730 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.799438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.799369 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk"] Oct 30 00:35:48.811036 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.811017 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:35:48.811201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.811182 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:35:48.817924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.817906 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:35:48.818017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.817928 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:35:48.818126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.818071 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:35:48.818126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.818076 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gqpv5\"" Oct 30 00:35:48.914912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.914894 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.914980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.914935 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.914980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.914972 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.915065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.915048 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.915102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.915070 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jzs\" (UniqueName: \"kubernetes.io/projected/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-kube-api-access-x8jzs\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.915102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.915089 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:48.962002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:48.961979 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:49.016077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016051 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jzs\" (UniqueName: \"kubernetes.io/projected/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-kube-api-access-x8jzs\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016081 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016107 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016166 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016185 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.016705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.016665 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.018437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.018410 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.018524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.018455 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.018582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.018565 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.018709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.018688 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.043472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.043449 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jzs\" (UniqueName: \"kubernetes.io/projected/8afdb479-e42f-47c4-bd9c-684b2d2a3b85-kube-api-access-x8jzs\") pod \"opendatahub-operator-controller-manager-6bcdb85766-jqwwk\" (UID: \"8afdb479-e42f-47c4-bd9c-684b2d2a3b85\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.078217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.078197 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:49.125000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.124811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:49.269644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.269562 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:49.469936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.469897 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk"] Oct 30 00:35:49.473164 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:35:49.473136 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afdb479_e42f_47c4_bd9c_684b2d2a3b85.slice/crio-bea7c6f52048e249e1afb8da54d8ea96d56ca0ac67ae5f0f2b081e04f668e77a WatchSource:0}: Error finding container bea7c6f52048e249e1afb8da54d8ea96d56ca0ac67ae5f0f2b081e04f668e77a: Status 404 returned error can't find the container with id bea7c6f52048e249e1afb8da54d8ea96d56ca0ac67ae5f0f2b081e04f668e77a Oct 30 00:35:49.483401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.483378 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:49.712168 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:49.712136 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" event={"ID":"8afdb479-e42f-47c4-bd9c-684b2d2a3b85","Type":"ContainerStarted","Data":"bea7c6f52048e249e1afb8da54d8ea96d56ca0ac67ae5f0f2b081e04f668e77a"} Oct 30 00:35:52.724985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:52.724953 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" event={"ID":"8afdb479-e42f-47c4-bd9c-684b2d2a3b85","Type":"ContainerStarted","Data":"207147a881a7ced8f8198962cf771185edf6d65d9f2df5f42a9f64e86bfbb10f"} Oct 30 00:35:53.729973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:53.729939 5938 generic.go:356] "Generic (PLEG): container finished" podID="8afdb479-e42f-47c4-bd9c-684b2d2a3b85" containerID="207147a881a7ced8f8198962cf771185edf6d65d9f2df5f42a9f64e86bfbb10f" exitCode=0 Oct 30 00:35:53.730430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:53.730021 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" event={"ID":"8afdb479-e42f-47c4-bd9c-684b2d2a3b85","Type":"ContainerDied","Data":"207147a881a7ced8f8198962cf771185edf6d65d9f2df5f42a9f64e86bfbb10f"} Oct 30 00:35:54.736700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:54.736666 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" event={"ID":"8afdb479-e42f-47c4-bd9c-684b2d2a3b85","Type":"ContainerStarted","Data":"8a502f82f0c9a2f86cade8d0f246d485fa46b1a00f9e9d0e59be8e1f396c6e77"} Oct 30 00:35:54.737101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:54.736817 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:35:54.825617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:54.825569 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" podStartSLOduration=3.210723428 podStartE2EDuration="6.825553596s" totalImagesPullingTime="3.176360118s" totalInitContainerRuntime="438.47005ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:54.813450012 +0000 UTC m=+207.053685124" watchObservedRunningTime="2026-10-30 00:35:54.825553596 +0000 UTC m=+207.065788700" Oct 30 00:35:56.487718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:56.487674 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:56.659999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:56.659964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:58.833266 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:35:58.833227 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:35:59.434619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:59.434579 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:35:59.607208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:35:59.607180 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:04.592896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:04.592848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:04.737833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:04.737801 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:05.743479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:05.743448 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-jqwwk" Oct 30 00:36:06.598157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.598120 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68"] Oct 30 00:36:06.603295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.603247 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.619521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.619492 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 00:36:06.620069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.620035 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 00:36:06.620196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.620143 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:36:06.631124 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.631100 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d66jw\"" Oct 30 00:36:06.631243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.631174 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 00:36:06.631296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.631241 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 00:36:06.631715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.631697 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68"] Oct 30 00:36:06.738709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.738681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/40cbce1e-d84a-4aee-877c-2a513b45a6d8-kube-api-access-rzvln\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.738827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.738743 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.738827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.738786 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.738927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.738885 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40cbce1e-d84a-4aee-877c-2a513b45a6d8-manager-config\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.839745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.839718 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.840177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.839745 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.840177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.839776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40cbce1e-d84a-4aee-877c-2a513b45a6d8-manager-config\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.840177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.839804 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/40cbce1e-d84a-4aee-877c-2a513b45a6d8-kube-api-access-rzvln\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.840435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.840414 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/40cbce1e-d84a-4aee-877c-2a513b45a6d8-manager-config\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.842347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.842320 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-metrics-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.842560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.842536 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40cbce1e-d84a-4aee-877c-2a513b45a6d8-cert\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.871304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.871275 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvln\" (UniqueName: \"kubernetes.io/projected/40cbce1e-d84a-4aee-877c-2a513b45a6d8-kube-api-access-rzvln\") pod \"lws-controller-manager-7cf8f946d-ndr68\" (UID: \"40cbce1e-d84a-4aee-877c-2a513b45a6d8\") " pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:06.918058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:06.918039 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:07.218354 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:07.218241 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68"] Oct 30 00:36:07.223532 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:36:07.223498 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cbce1e_d84a_4aee_877c_2a513b45a6d8.slice/crio-635486677e8b09e8e8fd00ad314ea7f208acfdf169e51e2397a402672452e214 WatchSource:0}: Error finding container 635486677e8b09e8e8fd00ad314ea7f208acfdf169e51e2397a402672452e214: Status 404 returned error can't find the container with id 635486677e8b09e8e8fd00ad314ea7f208acfdf169e51e2397a402672452e214 Oct 30 00:36:07.784112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:07.784079 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" event={"ID":"40cbce1e-d84a-4aee-877c-2a513b45a6d8","Type":"ContainerStarted","Data":"635486677e8b09e8e8fd00ad314ea7f208acfdf169e51e2397a402672452e214"} Oct 30 00:36:08.390677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.390638 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:08.544005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.543971 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:08.957811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.957781 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-w85dq"] Oct 30 00:36:08.960279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.960255 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:08.997347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.997319 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-w85dq"] Oct 30 00:36:08.999085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:08.999061 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:36:09.000585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.000565 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:36:09.009647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.009625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:09.015158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.015017 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 00:36:09.015158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.015032 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 00:36:09.015158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.015155 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jd954\"" Oct 30 00:36:09.059497 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.059469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2695552-6541-4a4b-90ab-c26a57b3c570-tls-certs\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.059618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.059531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2695552-6541-4a4b-90ab-c26a57b3c570-tmp\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.059672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.059650 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfg6\" (UniqueName: \"kubernetes.io/projected/b2695552-6541-4a4b-90ab-c26a57b3c570-kube-api-access-sjfg6\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.154820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.154789 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:09.160916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.160890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfg6\" (UniqueName: \"kubernetes.io/projected/b2695552-6541-4a4b-90ab-c26a57b3c570-kube-api-access-sjfg6\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.161028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.160930 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2695552-6541-4a4b-90ab-c26a57b3c570-tls-certs\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.161028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.160987 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2695552-6541-4a4b-90ab-c26a57b3c570-tmp\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.163499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.163480 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2695552-6541-4a4b-90ab-c26a57b3c570-tmp\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.164348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.164326 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2695552-6541-4a4b-90ab-c26a57b3c570-tls-certs\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.187478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.187449 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfg6\" (UniqueName: \"kubernetes.io/projected/b2695552-6541-4a4b-90ab-c26a57b3c570-kube-api-access-sjfg6\") pod \"kube-auth-proxy-b7df967f4-w85dq\" (UID: \"b2695552-6541-4a4b-90ab-c26a57b3c570\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.275338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.275277 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" Oct 30 00:36:09.578272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.578202 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-w85dq"] Oct 30 00:36:09.582155 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:36:09.582127 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2695552_6541_4a4b_90ab_c26a57b3c570.slice/crio-a9acc0acf57631b5f7f43b937b26c8baf2e109eaf92d266690b55c8348ba487d WatchSource:0}: Error finding container a9acc0acf57631b5f7f43b937b26c8baf2e109eaf92d266690b55c8348ba487d: Status 404 returned error can't find the container with id a9acc0acf57631b5f7f43b937b26c8baf2e109eaf92d266690b55c8348ba487d Oct 30 00:36:09.792555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:09.792514 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" event={"ID":"b2695552-6541-4a4b-90ab-c26a57b3c570","Type":"ContainerStarted","Data":"a9acc0acf57631b5f7f43b937b26c8baf2e109eaf92d266690b55c8348ba487d"} Oct 30 00:36:10.720346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:10.720269 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:10.916449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:10.916420 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:11.803043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:11.803003 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" event={"ID":"40cbce1e-d84a-4aee-877c-2a513b45a6d8","Type":"ContainerStarted","Data":"d89ee47d07d02558669ddb2d5172e0a6ee56a44b04438afbc908f87ed4c19504"} Oct 30 00:36:11.803459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:11.803115 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:11.843403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:11.843353 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" podStartSLOduration=2.283302428 podStartE2EDuration="5.84333695s" totalImagesPullingTime="3.560034522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:11.842344664 +0000 UTC m=+224.082579798" watchObservedRunningTime="2026-10-30 00:36:11.84333695 +0000 UTC m=+224.083572063" Oct 30 00:36:12.076858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:12.076830 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:12.570803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:12.570772 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:12.808393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:12.808362 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" event={"ID":"b2695552-6541-4a4b-90ab-c26a57b3c570","Type":"ContainerStarted","Data":"462cee014120acd516e973f49c5deb7501b88b8709256d97ccd3017cfa5a5d3b"} Oct 30 00:36:12.850508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:12.850422 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b7df967f4-w85dq" podStartSLOduration=2.35854653 podStartE2EDuration="4.850403761s" totalImagesPullingTime="2.491857231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:12.849705661 +0000 UTC m=+225.089940773" watchObservedRunningTime="2026-10-30 00:36:12.850403761 +0000 UTC m=+225.090638875" Oct 30 00:36:12.898989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:12.898941 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:13.219533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:13.219504 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:14.725379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:14.725343 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:14.929180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:14.929131 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:15.703466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:15.703432 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:15.860189 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:15.860156 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:16.127721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:16.127683 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:22.810381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:22.810349 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cf8f946d-ndr68" Oct 30 00:36:26.782765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:26.782726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:26.936227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:26.936190 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:28.833242 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:36:28.833205 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:36:34.369643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:36:34.369601 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:36:58.833708 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:36:58.833669 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:37:14.965655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:14.965575 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.139433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.139404 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.286279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.286206 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.330796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.330767 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-xjxz2"] Oct 30 00:37:15.333111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.333089 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.353373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.353349 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-xjxz2"] Oct 30 00:37:15.425551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.425521 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnhv\" (UniqueName: \"kubernetes.io/projected/3c55b307-9d39-4398-bf90-5dcaa5e22d15-kube-api-access-7fnhv\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.425685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.425566 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-cert\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.425685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.425650 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-metrics-certs\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.435063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.435036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.446521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.446503 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 00:37:15.446604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.446589 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5jjbg\"" Oct 30 00:37:15.446642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.446621 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 00:37:15.526397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.526374 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnhv\" (UniqueName: \"kubernetes.io/projected/3c55b307-9d39-4398-bf90-5dcaa5e22d15-kube-api-access-7fnhv\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.526483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.526415 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-cert\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.526483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.526452 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-metrics-certs\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.528831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.528811 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-cert\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.528922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.528854 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c55b307-9d39-4398-bf90-5dcaa5e22d15-metrics-certs\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.548215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.548165 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnhv\" (UniqueName: \"kubernetes.io/projected/3c55b307-9d39-4398-bf90-5dcaa5e22d15-kube-api-access-7fnhv\") pod \"odh-model-controller-f87fcf56c-xjxz2\" (UID: \"3c55b307-9d39-4398-bf90-5dcaa5e22d15\") " pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.608708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.608682 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.644833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.644810 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:15.772023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.771993 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:15.934328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:15.934303 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-xjxz2"] Oct 30 00:37:15.937908 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:37:15.937879 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c55b307_9d39_4398_bf90_5dcaa5e22d15.slice/crio-428585d1c4b1015ecc8486ee7022399f0bfc1c1c61952bb5533419622b0407c9 WatchSource:0}: Error finding container 428585d1c4b1015ecc8486ee7022399f0bfc1c1c61952bb5533419622b0407c9: Status 404 returned error can't find the container with id 428585d1c4b1015ecc8486ee7022399f0bfc1c1c61952bb5533419622b0407c9 Oct 30 00:37:16.031490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:16.031465 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" event={"ID":"3c55b307-9d39-4398-bf90-5dcaa5e22d15","Type":"ContainerStarted","Data":"428585d1c4b1015ecc8486ee7022399f0bfc1c1c61952bb5533419622b0407c9"} Oct 30 00:37:17.611455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:17.611417 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:17.750738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:17.750702 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:17.895524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:17.895442 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:18.032115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:18.032083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:19.044036 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:19.044007 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" event={"ID":"3c55b307-9d39-4398-bf90-5dcaa5e22d15","Type":"ContainerStarted","Data":"1af2d0e97dbc9c03af3712fdf9502fd9b438aa7694ea74cb733a17e88b296afd"} Oct 30 00:37:19.044418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:19.044136 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:19.084022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:19.083981 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" podStartSLOduration=1.698892922 podStartE2EDuration="4.083968249s" totalImagesPullingTime="2.385075327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:19.082821829 +0000 UTC m=+291.323056943" watchObservedRunningTime="2026-10-30 00:37:19.083968249 +0000 UTC m=+291.324203361" Oct 30 00:37:20.680344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:20.680304 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:27.904874 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:27.904846 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:37:27.905436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:27.905415 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:37:28.833450 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:37:28.833408 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:37:30.050353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:30.050312 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-xjxz2" Oct 30 00:37:34.497790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:37:34.497755 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:37:58.834724 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:37:58.834684 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:38:01.660257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:01.660215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:01.828932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:01.828905 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:12.483584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:12.483541 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:14.971526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:14.971485 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:16.580408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:16.580374 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:16.733528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:16.733489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:18.217416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:18.217388 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:18.686092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:18.686063 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:19.086982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:19.086903 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:19.505653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:19.505618 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:19.871788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:19.871757 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:20.035322 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:20.035293 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:22.943324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:22.943289 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:24.325256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.325219 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h"] Oct 30 00:38:24.327919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.327893 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:24.333848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.333816 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrwr\" (UniqueName: \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\") pod \"limitador-operator-controller-manager-856dd584c5-vg79h\" (UID: \"51887482-9a45-402e-9a38-af7e459ffee5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:24.362394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.362365 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:38:24.362515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.362413 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:38:24.362515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.362480 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vf9hz\"" Oct 30 00:38:24.363438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.363411 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h"] Oct 30 00:38:24.398934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.398912 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:24.435108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.435084 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrwr\" (UniqueName: \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\") pod \"limitador-operator-controller-manager-856dd584c5-vg79h\" (UID: \"51887482-9a45-402e-9a38-af7e459ffee5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:24.463853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.463829 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrwr\" (UniqueName: \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\") pod \"limitador-operator-controller-manager-856dd584c5-vg79h\" (UID: \"51887482-9a45-402e-9a38-af7e459ffee5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:24.640075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.640048 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:24.932320 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:24.932246 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h"] Oct 30 00:38:24.936504 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:38:24.936473 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51887482_9a45_402e_9a38_af7e459ffee5.slice/crio-55ace67ab0691ade9249f0e55c963623cede5eab472029d55f8aa5c7a198657d WatchSource:0}: Error finding container 55ace67ab0691ade9249f0e55c963623cede5eab472029d55f8aa5c7a198657d: Status 404 returned error can't find the container with id 55ace67ab0691ade9249f0e55c963623cede5eab472029d55f8aa5c7a198657d Oct 30 00:38:25.313847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:25.313772 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" event={"ID":"51887482-9a45-402e-9a38-af7e459ffee5","Type":"ContainerStarted","Data":"55ace67ab0691ade9249f0e55c963623cede5eab472029d55f8aa5c7a198657d"} Oct 30 00:38:25.885557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:25.885525 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.032561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:26.032521 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.279549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:26.279468 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:26.438278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:26.438243 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:28.333247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:28.333206 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" event={"ID":"51887482-9a45-402e-9a38-af7e459ffee5","Type":"ContainerStarted","Data":"7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a"} Oct 30 00:38:28.333784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:28.333494 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:28.373599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:28.373544 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" podStartSLOduration=1.9574165620000001 podStartE2EDuration="4.373527063s" totalImagesPullingTime="2.416110501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:28.372428094 +0000 UTC m=+360.612663208" watchObservedRunningTime="2026-10-30 00:38:28.373527063 +0000 UTC m=+360.613762177" Oct 30 00:38:28.833376 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:28.833335 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:38:29.678833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:29.678802 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:29.821550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:29.821524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:30.859742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:30.859698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:31.031755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:31.031703 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.627335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:34.627301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.788246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:34.788215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:34.934929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:34.934840 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:39.340220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:39.340138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:41.744162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:41.744112 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:41.886448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:41.886417 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:43.312966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.312927 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h"] Oct 30 00:38:43.313451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.313148 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" podUID="51887482-9a45-402e-9a38-af7e459ffee5" containerName="manager" containerID="cri-o://7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a" gracePeriod=2 Oct 30 00:38:43.319457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.319431 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h"] Oct 30 00:38:43.336061 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:43.336021 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-vg79h\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="51887482-9a45-402e-9a38-af7e459ffee5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:43.336447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.336420 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52"] Oct 30 00:38:43.336964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.336935 5938 cpu_manager.go:414] "containerMap: removing container" podUID="51887482-9a45-402e-9a38-af7e459ffee5" containerName="manager" Oct 30 00:38:43.336964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.336955 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51887482-9a45-402e-9a38-af7e459ffee5" containerName="manager" Oct 30 00:38:43.337107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.337078 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="51887482-9a45-402e-9a38-af7e459ffee5" containerName="manager" Oct 30 00:38:43.340176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.340157 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:43.356786 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:43.355850 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-vg79h\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="51887482-9a45-402e-9a38-af7e459ffee5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:43.356786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.356729 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52"] Oct 30 00:38:43.380421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.380390 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgg9\" (UniqueName: \"kubernetes.io/projected/8098738c-5d92-4b27-9196-c0071d6590e4-kube-api-access-qlgg9\") pod \"limitador-operator-controller-manager-856dd584c5-n6n52\" (UID: \"8098738c-5d92-4b27-9196-c0071d6590e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:43.481963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.481931 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgg9\" (UniqueName: \"kubernetes.io/projected/8098738c-5d92-4b27-9196-c0071d6590e4-kube-api-access-qlgg9\") pod \"limitador-operator-controller-manager-856dd584c5-n6n52\" (UID: \"8098738c-5d92-4b27-9196-c0071d6590e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:43.505086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.505056 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgg9\" (UniqueName: \"kubernetes.io/projected/8098738c-5d92-4b27-9196-c0071d6590e4-kube-api-access-qlgg9\") pod \"limitador-operator-controller-manager-856dd584c5-n6n52\" (UID: \"8098738c-5d92-4b27-9196-c0071d6590e4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:43.560079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.560047 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:43.573071 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:43.573020 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-vg79h\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="51887482-9a45-402e-9a38-af7e459ffee5" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:43.682961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.682936 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\" (UniqueName: \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\") pod \"51887482-9a45-402e-9a38-af7e459ffee5\" (UID: \"51887482-9a45-402e-9a38-af7e459ffee5\") " Oct 30 00:38:43.685052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.685023 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr" pod "51887482-9a45-402e-9a38-af7e459ffee5" (UID: "51887482-9a45-402e-9a38-af7e459ffee5"). InnerVolumeSpecName "kube-api-access-rsrwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:38:43.703196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.703165 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:43.784378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.784353 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrwr\" (UniqueName: \"kubernetes.io/projected/51887482-9a45-402e-9a38-af7e459ffee5-kube-api-access-rsrwr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:38:43.872932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:43.872904 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51887482-9a45-402e-9a38-af7e459ffee5" path="/var/lib/kubelet/pods/51887482-9a45-402e-9a38-af7e459ffee5/volumes" Oct 30 00:38:44.019836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.019799 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52"] Oct 30 00:38:44.023931 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:38:44.023903 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8098738c_5d92_4b27_9196_c0071d6590e4.slice/crio-75bcc0c2034141678a6d4a668a21d1d9c48bf795211105195d802b368ee78579 WatchSource:0}: Error finding container 75bcc0c2034141678a6d4a668a21d1d9c48bf795211105195d802b368ee78579: Status 404 returned error can't find the container with id 75bcc0c2034141678a6d4a668a21d1d9c48bf795211105195d802b368ee78579 Oct 30 00:38:44.412629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.412597 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" event={"ID":"8098738c-5d92-4b27-9196-c0071d6590e4","Type":"ContainerStarted","Data":"aeb10a4f6678ab2d050c968d824e91b91edced8ec25580bc2fb0bf3c7ba49e6c"} Oct 30 00:38:44.413021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.412660 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" event={"ID":"8098738c-5d92-4b27-9196-c0071d6590e4","Type":"ContainerStarted","Data":"75bcc0c2034141678a6d4a668a21d1d9c48bf795211105195d802b368ee78579"} Oct 30 00:38:44.413021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.412693 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:44.413964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.413943 5938 generic.go:356] "Generic (PLEG): container finished" podID="51887482-9a45-402e-9a38-af7e459ffee5" containerID="7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a" exitCode=0 Oct 30 00:38:44.414058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.413985 5938 scope.go:122] "RemoveContainer" containerID="7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a" Oct 30 00:38:44.414058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.413995 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-vg79h" Oct 30 00:38:44.421336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.421319 5938 scope.go:122] "RemoveContainer" containerID="7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a" Oct 30 00:38:44.421589 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:44.421570 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a\": container with ID starting with 7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a not found: ID does not exist" containerID="7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a" Oct 30 00:38:44.421650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.421595 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a"} err="failed to get container status \"7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a\": rpc error: code = NotFound desc = could not find container \"7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a\": container with ID starting with 7729c8665a3e5dd5e45ac7e6016603e80c7c118a5740bb876fcc9703eaef2c9a not found: ID does not exist" Oct 30 00:38:44.453356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.453317 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" podStartSLOduration=1.45330606 podStartE2EDuration="1.45330606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:44.452259613 +0000 UTC m=+376.692494728" watchObservedRunningTime="2026-10-30 00:38:44.45330606 +0000 UTC m=+376.693541172" Oct 30 00:38:44.881071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:44.881041 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:45.039246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:45.039215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:46.602706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:46.602645 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:46.758857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:46.758827 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:48.125903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:48.125845 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:48.260020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:48.259989 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:53.950775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:53.950747 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:54.121086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:54.121053 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:55.421948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:55.421919 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-n6n52" Oct 30 00:38:58.405773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:58.405732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:58.549515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:38:58.549484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:38:58.832942 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:38:58.832847 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:39:00.977260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:00.977228 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:01.118569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:01.118536 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:01.492469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:01.492438 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:01.669461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:01.669435 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:13.034975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:13.034931 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:13.217011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:13.216961 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:28.832931 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:39:28.832885 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:39:30.547205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:30.547167 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.692613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:30.692576 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:30.837360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:30.837283 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:31.386201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.386166 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:31.389426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.389399 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:31.406142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.406109 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:31.413033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.413012 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4frwg\"" Oct 30 00:39:31.512746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.512697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:31.525913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.525887 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crz6h\" (UniqueName: \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\") pod \"authorino-57f7f8659f-5zrgc\" (UID: \"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551\") " pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:31.627229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.627205 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crz6h\" (UniqueName: \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\") pod \"authorino-57f7f8659f-5zrgc\" (UID: \"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551\") " pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:31.650391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.650333 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crz6h\" (UniqueName: \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\") pod \"authorino-57f7f8659f-5zrgc\" (UID: \"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551\") " pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:31.672983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.672956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:31.700945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.700924 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:31.992159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:31.992082 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:31.995481 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:39:31.995456 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3afb2ed0_5ae4_43c4_acf0_42ca2bb95551.slice/crio-4f614f0abfc540513812b80ce21b52bf8b5cad77c55e5aef74efe480acd65df2 WatchSource:0}: Error finding container 4f614f0abfc540513812b80ce21b52bf8b5cad77c55e5aef74efe480acd65df2: Status 404 returned error can't find the container with id 4f614f0abfc540513812b80ce21b52bf8b5cad77c55e5aef74efe480acd65df2 Oct 30 00:39:32.019473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.019449 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:39:32.022418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.022400 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:39:32.041638 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.041616 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:39:32.131462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.131435 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d97v\" (UniqueName: \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\") pod \"authorino-56c6c9ddd5-xqkwt\" (UID: \"b33b0432-c93d-4ae8-8314-3accef38576a\") " pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:39:32.231767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.231743 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d97v\" (UniqueName: \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\") pod \"authorino-56c6c9ddd5-xqkwt\" (UID: \"b33b0432-c93d-4ae8-8314-3accef38576a\") " pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:39:32.254393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.254336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d97v\" (UniqueName: \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\") pod \"authorino-56c6c9ddd5-xqkwt\" (UID: \"b33b0432-c93d-4ae8-8314-3accef38576a\") " pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:39:32.332972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.332947 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:39:32.612279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.612240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" event={"ID":"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551","Type":"ContainerStarted","Data":"4f614f0abfc540513812b80ce21b52bf8b5cad77c55e5aef74efe480acd65df2"} Oct 30 00:39:32.625042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:32.625014 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:39:32.629055 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:39:32.629029 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb33b0432_c93d_4ae8_8314_3accef38576a.slice/crio-9289ea929f4fc3c4ec391448fa0fa5f4d38b5299d53da8b4b52b1f9fdf743a2a WatchSource:0}: Error finding container 9289ea929f4fc3c4ec391448fa0fa5f4d38b5299d53da8b4b52b1f9fdf743a2a: Status 404 returned error can't find the container with id 9289ea929f4fc3c4ec391448fa0fa5f4d38b5299d53da8b4b52b1f9fdf743a2a Oct 30 00:39:33.616807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:33.616771 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" event={"ID":"b33b0432-c93d-4ae8-8314-3accef38576a","Type":"ContainerStarted","Data":"9289ea929f4fc3c4ec391448fa0fa5f4d38b5299d53da8b4b52b1f9fdf743a2a"} Oct 30 00:39:33.618149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:33.618124 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" event={"ID":"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551","Type":"ContainerStarted","Data":"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad"} Oct 30 00:39:33.656592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:33.656541 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" podStartSLOduration=1.173180413 podStartE2EDuration="2.656527449s" totalImagesPullingTime="1.483347036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:33.655115454 +0000 UTC m=+425.895350568" watchObservedRunningTime="2026-10-30 00:39:33.656527449 +0000 UTC m=+425.896762562" Oct 30 00:39:34.624769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:34.624732 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" event={"ID":"b33b0432-c93d-4ae8-8314-3accef38576a","Type":"ContainerStarted","Data":"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e"} Oct 30 00:39:34.662388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:34.662333 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" podStartSLOduration=2.237068272 podStartE2EDuration="3.66231926s" totalImagesPullingTime="1.425250988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:34.661626862 +0000 UTC m=+426.901861975" watchObservedRunningTime="2026-10-30 00:39:34.66231926 +0000 UTC m=+426.902554372" Oct 30 00:39:34.690553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:34.690522 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:34.742622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:34.742596 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:35.629213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:35.629174 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" podUID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" containerName="authorino" containerID="cri-o://5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad" gracePeriod=30 Oct 30 00:39:35.869887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:35.869847 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:35.961735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:35.961677 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\" (UniqueName: \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\") pod \"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551\" (UID: \"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551\") " Oct 30 00:39:35.963599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:35.963574 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h" pod "3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" (UID: "3afb2ed0-5ae4-43c4-acf0-42ca2bb95551"). InnerVolumeSpecName "kube-api-access-crz6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:39:36.062629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.062604 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crz6h\" (UniqueName: \"kubernetes.io/projected/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551-kube-api-access-crz6h\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:39:36.634736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.634698 5938 generic.go:356] "Generic (PLEG): container finished" podID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" containerID="5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad" exitCode=0 Oct 30 00:39:36.634877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.634750 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" Oct 30 00:39:36.634877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.634791 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" event={"ID":"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551","Type":"ContainerDied","Data":"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad"} Oct 30 00:39:36.634877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.634840 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5zrgc" event={"ID":"3afb2ed0-5ae4-43c4-acf0-42ca2bb95551","Type":"ContainerDied","Data":"4f614f0abfc540513812b80ce21b52bf8b5cad77c55e5aef74efe480acd65df2"} Oct 30 00:39:36.634996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.634880 5938 scope.go:122] "RemoveContainer" containerID="5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad" Oct 30 00:39:36.644333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.644308 5938 scope.go:122] "RemoveContainer" containerID="5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad" Oct 30 00:39:36.644621 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:39:36.644603 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad\": container with ID starting with 5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad not found: ID does not exist" containerID="5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad" Oct 30 00:39:36.644668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.644629 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad"} err="failed to get container status \"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad\": rpc error: code = NotFound desc = could not find container \"5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad\": container with ID starting with 5cf4d972e7ea8004794b0190cda5499b9a2b574467af5eb41ac563ab6948c2ad not found: ID does not exist" Oct 30 00:39:36.682723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.682688 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:36.686606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:36.686583 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5zrgc"] Oct 30 00:39:37.878766 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:37.878728 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" path="/var/lib/kubelet/pods/3afb2ed0-5ae4-43c4-acf0-42ca2bb95551/volumes" Oct 30 00:39:47.373756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:47.373723 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:47.526533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:47.526496 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:48.090998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:39:48.090967 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:39:58.834804 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:39:58.834766 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:40:00.555376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.555300 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:00.705100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.705070 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:00.807737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.807658 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrvnw"] Oct 30 00:40:00.808013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.807998 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" containerName="authorino" Oct 30 00:40:00.808062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.808014 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" containerName="authorino" Oct 30 00:40:00.808102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.808082 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afb2ed0-5ae4-43c4-acf0-42ca2bb95551" containerName="authorino" Oct 30 00:40:00.810325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.810308 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:00.828016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.827992 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrvnw"] Oct 30 00:40:00.946598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:00.946569 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp6h\" (UniqueName: \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\") pod \"authorino-6f6944bcdd-lrvnw\" (UID: \"28386922-b35c-401e-8355-87e679ea3cd2\") " pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:01.011668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.011645 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrvnw"] Oct 30 00:40:01.011819 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:40:01.011800 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tzp6h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" podUID="28386922-b35c-401e-8355-87e679ea3cd2" Oct 30 00:40:01.040026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.040003 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bln7h"] Oct 30 00:40:01.042226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.042208 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.047192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.047153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp6h\" (UniqueName: \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\") pod \"authorino-6f6944bcdd-lrvnw\" (UID: \"28386922-b35c-401e-8355-87e679ea3cd2\") " pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:01.056552 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.056533 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 00:40:01.061848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.061794 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bln7h"] Oct 30 00:40:01.070996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.070978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp6h\" (UniqueName: \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\") pod \"authorino-6f6944bcdd-lrvnw\" (UID: \"28386922-b35c-401e-8355-87e679ea3cd2\") " pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:01.146082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.146047 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bln7h"] Oct 30 00:40:01.146216 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:40:01.146199 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sltln tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-bln7h" podUID="be22ae69-ec84-44b7-ab02-91ac1f8b0930" Oct 30 00:40:01.148497 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.148478 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.148654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.148511 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltln\" (UniqueName: \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.174304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.174279 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:40:01.177151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.177133 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.195329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.195310 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:40:01.249714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.249691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.249817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.249731 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltln\" (UniqueName: \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.252120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.252100 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.272007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.271984 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltln\" (UniqueName: \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\") pod \"authorino-6889cbd8c4-bln7h\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.350802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.350750 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.350802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.350779 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgws\" (UniqueName: \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.451330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.451288 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.451464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.451349 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgws\" (UniqueName: \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.453711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.453683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.473703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.473674 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgws\" (UniqueName: \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\") pod \"authorino-6558f48958-22jmp\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.487465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.487447 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:40:01.738732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.738703 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:01.739098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.738703 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.745332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.745308 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:01.748245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.748225 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:01.775591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.775541 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:40:01.779325 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:40:01.779303 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46073246_fb65_46cd_9648_ffc11a6cb904.slice/crio-97a0cd4dadbb3ee598e907a4efcea3ffea9d3cb9cf5df1679ed33ae4304e6fdd WatchSource:0}: Error finding container 97a0cd4dadbb3ee598e907a4efcea3ffea9d3cb9cf5df1679ed33ae4304e6fdd: Status 404 returned error can't find the container with id 97a0cd4dadbb3ee598e907a4efcea3ffea9d3cb9cf5df1679ed33ae4304e6fdd Oct 30 00:40:01.854783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.854762 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\" (UniqueName: \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\") pod \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " Oct 30 00:40:01.854910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.854804 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\" (UniqueName: \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\") pod \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\" (UID: \"be22ae69-ec84-44b7-ab02-91ac1f8b0930\") " Oct 30 00:40:01.854910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.854853 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\" (UniqueName: \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\") pod \"28386922-b35c-401e-8355-87e679ea3cd2\" (UID: \"28386922-b35c-401e-8355-87e679ea3cd2\") " Oct 30 00:40:01.856804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.856772 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln" pod "be22ae69-ec84-44b7-ab02-91ac1f8b0930" (UID: "be22ae69-ec84-44b7-ab02-91ac1f8b0930"). InnerVolumeSpecName "kube-api-access-sltln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:01.856898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.856846 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h" pod "28386922-b35c-401e-8355-87e679ea3cd2" (UID: "28386922-b35c-401e-8355-87e679ea3cd2"). InnerVolumeSpecName "kube-api-access-tzp6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:01.856974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.856928 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert" pod "be22ae69-ec84-44b7-ab02-91ac1f8b0930" (UID: "be22ae69-ec84-44b7-ab02-91ac1f8b0930"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:40:01.956225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.956200 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp6h\" (UniqueName: \"kubernetes.io/projected/28386922-b35c-401e-8355-87e679ea3cd2-kube-api-access-tzp6h\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:40:01.956225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.956224 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sltln\" (UniqueName: \"kubernetes.io/projected/be22ae69-ec84-44b7-ab02-91ac1f8b0930-kube-api-access-sltln\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:40:01.956353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:01.956234 5938 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be22ae69-ec84-44b7-ab02-91ac1f8b0930-tls-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:40:02.745326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.745279 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6558f48958-22jmp" event={"ID":"46073246-fb65-46cd-9648-ffc11a6cb904","Type":"ContainerStarted","Data":"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19"} Oct 30 00:40:02.745830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.745335 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bln7h" Oct 30 00:40:02.745830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.745342 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6558f48958-22jmp" event={"ID":"46073246-fb65-46cd-9648-ffc11a6cb904","Type":"ContainerStarted","Data":"97a0cd4dadbb3ee598e907a4efcea3ffea9d3cb9cf5df1679ed33ae4304e6fdd"} Oct 30 00:40:02.745830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.745350 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrvnw" Oct 30 00:40:02.798557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.798512 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6558f48958-22jmp" podStartSLOduration=1.401460261 podStartE2EDuration="1.798497874s" totalImagesPullingTime="397.037613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:02.797179519 +0000 UTC m=+455.037414632" watchObservedRunningTime="2026-10-30 00:40:02.798497874 +0000 UTC m=+455.038732986" Oct 30 00:40:02.829276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.829246 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:40:02.829439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.829420 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" podUID="b33b0432-c93d-4ae8-8314-3accef38576a" containerName="authorino" containerID="cri-o://984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e" gracePeriod=30 Oct 30 00:40:02.854695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.854673 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrvnw"] Oct 30 00:40:02.860778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.860751 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrvnw"] Oct 30 00:40:02.917365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.917340 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bln7h"] Oct 30 00:40:02.921415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:02.921386 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bln7h"] Oct 30 00:40:03.049076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.049053 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:40:03.166847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.166818 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\" (UniqueName: \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\") pod \"b33b0432-c93d-4ae8-8314-3accef38576a\" (UID: \"b33b0432-c93d-4ae8-8314-3accef38576a\") " Oct 30 00:40:03.168774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.168751 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v" pod "b33b0432-c93d-4ae8-8314-3accef38576a" (UID: "b33b0432-c93d-4ae8-8314-3accef38576a"). InnerVolumeSpecName "kube-api-access-2d97v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:40:03.268037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.268012 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d97v\" (UniqueName: \"kubernetes.io/projected/b33b0432-c93d-4ae8-8314-3accef38576a-kube-api-access-2d97v\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:40:03.750071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.750043 5938 generic.go:356] "Generic (PLEG): container finished" podID="b33b0432-c93d-4ae8-8314-3accef38576a" containerID="984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e" exitCode=0 Oct 30 00:40:03.750402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.750086 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" Oct 30 00:40:03.750402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.750134 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" event={"ID":"b33b0432-c93d-4ae8-8314-3accef38576a","Type":"ContainerDied","Data":"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e"} Oct 30 00:40:03.750402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.750179 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xqkwt" event={"ID":"b33b0432-c93d-4ae8-8314-3accef38576a","Type":"ContainerDied","Data":"9289ea929f4fc3c4ec391448fa0fa5f4d38b5299d53da8b4b52b1f9fdf743a2a"} Oct 30 00:40:03.750402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.750198 5938 scope.go:122] "RemoveContainer" containerID="984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e" Oct 30 00:40:03.758649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.758633 5938 scope.go:122] "RemoveContainer" containerID="984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e" Oct 30 00:40:03.758931 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:40:03.758913 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e\": container with ID starting with 984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e not found: ID does not exist" containerID="984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e" Oct 30 00:40:03.759004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.758939 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e"} err="failed to get container status \"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e\": rpc error: code = NotFound desc = could not find container \"984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e\": container with ID starting with 984b6a879249ccc92db7e0a2ec492b39580a4add850346c7b1f8d59873242c2e not found: ID does not exist" Oct 30 00:40:03.799613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.799594 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:40:03.801832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.801813 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xqkwt"] Oct 30 00:40:03.874108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.874086 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28386922-b35c-401e-8355-87e679ea3cd2" path="/var/lib/kubelet/pods/28386922-b35c-401e-8355-87e679ea3cd2/volumes" Oct 30 00:40:03.874295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.874283 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33b0432-c93d-4ae8-8314-3accef38576a" path="/var/lib/kubelet/pods/b33b0432-c93d-4ae8-8314-3accef38576a/volumes" Oct 30 00:40:03.874581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:03.874570 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be22ae69-ec84-44b7-ab02-91ac1f8b0930" path="/var/lib/kubelet/pods/be22ae69-ec84-44b7-ab02-91ac1f8b0930/volumes" Oct 30 00:40:09.858049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:09.857971 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:09.998383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:09.998347 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:10.136679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:10.136608 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:13.476678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:13.476642 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:13.628096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:13.628067 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:17.091269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:17.091229 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:17.248940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:17.248908 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:28.833771 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:40:28.833729 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:40:34.973558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:34.973514 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:45.972772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:45.972732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:46.153701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:46.153676 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:47.489857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:47.489821 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:47.656021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:47.655988 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:58.713818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:58.713773 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:40:58.833124 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:40:58.833088 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:40:58.852768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:40:58.852745 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:28.832994 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:41:28.832958 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:41:30.829881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:30.829834 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:30.991764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:30.991727 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:34.986168 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:34.986102 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:47.460475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.460385 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv"] Oct 30 00:41:47.460935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.460898 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b33b0432-c93d-4ae8-8314-3accef38576a" containerName="authorino" Oct 30 00:41:47.460935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.460917 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33b0432-c93d-4ae8-8314-3accef38576a" containerName="authorino" Oct 30 00:41:47.461042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.461025 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33b0432-c93d-4ae8-8314-3accef38576a" containerName="authorino" Oct 30 00:41:47.464517 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.464494 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.479546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.479522 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 00:41:47.491086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.491047 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 00:41:47.491086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.491075 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 00:41:47.491264 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.491168 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dkdpm\"" Oct 30 00:41:47.492045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.492025 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv"] Oct 30 00:41:47.543726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.543691 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzcb\" (UniqueName: \"kubernetes.io/projected/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-kube-api-access-lrzcb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.543930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.543804 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.543930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.543857 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.543930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.543910 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.544115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.544008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.544115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.544038 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.553927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.553900 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:47.645370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645343 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645387 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645497 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzcb\" (UniqueName: \"kubernetes.io/projected/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-kube-api-access-lrzcb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645571 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645742 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.645947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645928 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.646014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.645986 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.647711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.647685 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.647856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.647840 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.667630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.667610 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzcb\" (UniqueName: \"kubernetes.io/projected/111fbfd3-f42e-4d29-b107-65ddfa99c0a4-kube-api-access-lrzcb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv\" (UID: \"111fbfd3-f42e-4d29-b107-65ddfa99c0a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:47.778555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:47.778495 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:48.072440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:48.072366 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv"] Oct 30 00:41:48.075635 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:41:48.075608 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111fbfd3_f42e_4d29_b107_65ddfa99c0a4.slice/crio-14e4d13fdfca64e27029516d0aa8a2aa01d7d04b6dd5201e83e9f58d0be40a07 WatchSource:0}: Error finding container 14e4d13fdfca64e27029516d0aa8a2aa01d7d04b6dd5201e83e9f58d0be40a07: Status 404 returned error can't find the container with id 14e4d13fdfca64e27029516d0aa8a2aa01d7d04b6dd5201e83e9f58d0be40a07 Oct 30 00:41:48.076836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:48.076821 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:41:48.175783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:48.175748 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" event={"ID":"111fbfd3-f42e-4d29-b107-65ddfa99c0a4","Type":"ContainerStarted","Data":"14e4d13fdfca64e27029516d0aa8a2aa01d7d04b6dd5201e83e9f58d0be40a07"} Oct 30 00:41:50.186384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:50.186345 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" event={"ID":"111fbfd3-f42e-4d29-b107-65ddfa99c0a4","Type":"ContainerStarted","Data":"32d73fc6faf614f4387d3aa90bc94c0c012b0664aa05bcd02d97da48e679feac"} Oct 30 00:41:50.228329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:50.228283 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" podStartSLOduration=1.985613121 podStartE2EDuration="3.228269581s" totalImagesPullingTime="1.24265646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:50.227160043 +0000 UTC m=+562.467395157" watchObservedRunningTime="2026-10-30 00:41:50.228269581 +0000 UTC m=+562.468504690" Oct 30 00:41:50.485822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:50.485750 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:50.779424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:50.779353 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:51.739450 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.739413 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d"] Oct 30 00:41:51.743015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.742991 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.768519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.768482 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 00:41:51.769707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.769687 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d"] Oct 30 00:41:51.786454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786431 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.786577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786464 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.786577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786484 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.786577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786501 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.786710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786636 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.786710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.786685 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qf5v\" (UniqueName: \"kubernetes.io/projected/5826cc58-3128-42cc-b1e6-06847d5ed3b2-kube-api-access-9qf5v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.882165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.882135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:51.887261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887233 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qf5v\" (UniqueName: \"kubernetes.io/projected/5826cc58-3128-42cc-b1e6-06847d5ed3b2-kube-api-access-9qf5v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887292 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887324 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887351 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887490 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887722 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887803 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.887854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.887802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.889882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.889838 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5826cc58-3128-42cc-b1e6-06847d5ed3b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.890038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.890019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5826cc58-3128-42cc-b1e6-06847d5ed3b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:51.909551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:51.909530 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qf5v\" (UniqueName: \"kubernetes.io/projected/5826cc58-3128-42cc-b1e6-06847d5ed3b2-kube-api-access-9qf5v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d\" (UID: \"5826cc58-3128-42cc-b1e6-06847d5ed3b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:52.056006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:52.055914 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:52.352798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:52.352771 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d"] Oct 30 00:41:52.355685 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:41:52.355657 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5826cc58_3128_42cc_b1e6_06847d5ed3b2.slice/crio-754b49ade665342689edda4fab2c69da53cc5c19809b6c7d15ac61138cf75920 WatchSource:0}: Error finding container 754b49ade665342689edda4fab2c69da53cc5c19809b6c7d15ac61138cf75920: Status 404 returned error can't find the container with id 754b49ade665342689edda4fab2c69da53cc5c19809b6c7d15ac61138cf75920 Oct 30 00:41:53.201878 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:53.201836 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" event={"ID":"5826cc58-3128-42cc-b1e6-06847d5ed3b2","Type":"ContainerStarted","Data":"e2c2807d15aa78d2fa2020e7f35508bc6b8fef54f7001d7ffe90f9d3a5cc0284"} Oct 30 00:41:53.202360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:53.201908 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" event={"ID":"5826cc58-3128-42cc-b1e6-06847d5ed3b2","Type":"ContainerStarted","Data":"754b49ade665342689edda4fab2c69da53cc5c19809b6c7d15ac61138cf75920"} Oct 30 00:41:53.244241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:53.244185 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podStartSLOduration=2.059062471 podStartE2EDuration="2.244166871s" totalImagesPullingTime="185.1044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:53.241900594 +0000 UTC m=+565.482135708" watchObservedRunningTime="2026-10-30 00:41:53.244166871 +0000 UTC m=+565.484401984" Oct 30 00:41:54.056597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:54.056534 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:41:54.137478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:54.137448 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:55.847805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.847767 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb"] Oct 30 00:41:55.851187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.851159 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.882646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.882619 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 00:41:55.883653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.883629 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb"] Oct 30 00:41:55.936800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.936770 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.937009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.936839 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8099822-8f87-43de-9860-0d34402a1669-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.937081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.937021 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.937081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.937055 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pxn\" (UniqueName: \"kubernetes.io/projected/f8099822-8f87-43de-9860-0d34402a1669-kube-api-access-s9pxn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.937220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.937178 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.937325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.937295 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:55.952784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:55.952755 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:56.038452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038414 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038465 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8099822-8f87-43de-9860-0d34402a1669-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038515 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pxn\" (UniqueName: \"kubernetes.io/projected/f8099822-8f87-43de-9860-0d34402a1669-kube-api-access-s9pxn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038571 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038608 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.038944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038904 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.039016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.038952 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.039068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.039019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.040830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.040806 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8099822-8f87-43de-9860-0d34402a1669-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.041143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.041125 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8099822-8f87-43de-9860-0d34402a1669-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.061335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.061303 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pxn\" (UniqueName: \"kubernetes.io/projected/f8099822-8f87-43de-9860-0d34402a1669-kube-api-access-s9pxn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jrvcb\" (UID: \"f8099822-8f87-43de-9860-0d34402a1669\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.167134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.167103 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:56.462967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:56.462885 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb"] Oct 30 00:41:56.465794 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:41:56.465763 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8099822_8f87_43de_9860_0d34402a1669.slice/crio-b58d77df306420e630a5f08585af226098d08937f5740d486b0dbb66f77aac6b WatchSource:0}: Error finding container b58d77df306420e630a5f08585af226098d08937f5740d486b0dbb66f77aac6b: Status 404 returned error can't find the container with id b58d77df306420e630a5f08585af226098d08937f5740d486b0dbb66f77aac6b Oct 30 00:41:57.229903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.229851 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" event={"ID":"f8099822-8f87-43de-9860-0d34402a1669","Type":"ContainerStarted","Data":"f93038e62b7e3d748fdaac941bafa2c2d969d1ca3b05b062dc63475ff53c0cbd"} Oct 30 00:41:57.230268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.229943 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" event={"ID":"f8099822-8f87-43de-9860-0d34402a1669","Type":"ContainerStarted","Data":"b58d77df306420e630a5f08585af226098d08937f5740d486b0dbb66f77aac6b"} Oct 30 00:41:57.272430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.272383 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" podStartSLOduration=2.093361516 podStartE2EDuration="2.27236858s" totalImagesPullingTime="179.007064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:57.271370332 +0000 UTC m=+569.511605468" watchObservedRunningTime="2026-10-30 00:41:57.27236858 +0000 UTC m=+569.512603692" Oct 30 00:41:57.337227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.337201 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg"] Oct 30 00:41:57.340568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.340547 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.355023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.355001 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 00:41:57.359720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.359694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg"] Oct 30 00:41:57.430613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.430588 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:41:57.449099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449076 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp944\" (UniqueName: \"kubernetes.io/projected/8a59a208-000a-448d-9ca1-6042908e69bb-kube-api-access-jp944\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.449199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449116 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a59a208-000a-448d-9ca1-6042908e69bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.449199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449136 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.449316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449294 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.449371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449358 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.449413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.449386 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550315 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp944\" (UniqueName: \"kubernetes.io/projected/8a59a208-000a-448d-9ca1-6042908e69bb-kube-api-access-jp944\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550370 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a59a208-000a-448d-9ca1-6042908e69bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550408 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550461 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550489 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.550991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.551088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.550971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.551088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.551013 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.552789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.552770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a59a208-000a-448d-9ca1-6042908e69bb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.553091 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.553069 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a59a208-000a-448d-9ca1-6042908e69bb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.572670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.572645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp944\" (UniqueName: \"kubernetes.io/projected/8a59a208-000a-448d-9ca1-6042908e69bb-kube-api-access-jp944\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wpnlg\" (UID: \"8a59a208-000a-448d-9ca1-6042908e69bb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.655631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.655603 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:57.779297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.779266 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:57.791426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.791409 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:57.948526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:57.948501 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg"] Oct 30 00:41:57.951583 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:41:57.951557 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a59a208_000a_448d_9ca1_6042908e69bb.slice/crio-b54fb77c0c6d782fe8f7f33ca0cfc311ec7fc0f872b5dd12c9468fdd37c3be39 WatchSource:0}: Error finding container b54fb77c0c6d782fe8f7f33ca0cfc311ec7fc0f872b5dd12c9468fdd37c3be39: Status 404 returned error can't find the container with id b54fb77c0c6d782fe8f7f33ca0cfc311ec7fc0f872b5dd12c9468fdd37c3be39 Oct 30 00:41:58.167573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.167551 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:41:58.235641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.235570 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" event={"ID":"8a59a208-000a-448d-9ca1-6042908e69bb","Type":"ContainerStarted","Data":"5291a9eca75d5f43f6d07053f1a9438cf469b97286d847c9e265fea81715e92a"} Oct 30 00:41:58.235641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.235628 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" event={"ID":"8a59a208-000a-448d-9ca1-6042908e69bb","Type":"ContainerStarted","Data":"b54fb77c0c6d782fe8f7f33ca0cfc311ec7fc0f872b5dd12c9468fdd37c3be39"} Oct 30 00:41:58.247725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.247701 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv" Oct 30 00:41:58.276898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.276739 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" podStartSLOduration=1.106120538 podStartE2EDuration="1.276722642s" totalImagesPullingTime="170.602104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:58.27607518 +0000 UTC m=+570.516310320" watchObservedRunningTime="2026-10-30 00:41:58.276722642 +0000 UTC m=+570.516957750" Oct 30 00:41:58.656266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:41:58.656227 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:41:58.835637 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:41:58.835599 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:42:00.162273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:00.162243 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:02.056597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:02.056561 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:42:02.069046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:02.069022 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:42:02.271100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:02.271073 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:42:05.549442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:05.549412 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:05.702664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:05.702635 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:05.897763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:05.897734 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-797fb9856f-8rq9w"] Oct 30 00:42:05.901125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:05.901103 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:05.919067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:05.919043 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797fb9856f-8rq9w"] Oct 30 00:42:06.046056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.046025 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpb2\" (UniqueName: \"kubernetes.io/projected/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-kube-api-access-xbpb2\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.046247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.046075 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-tls-cert\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.146781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.146752 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-tls-cert\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.146926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.146817 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpb2\" (UniqueName: \"kubernetes.io/projected/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-kube-api-access-xbpb2\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.149204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.149150 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-tls-cert\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.167567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.167541 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:42:06.170021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.169998 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpb2\" (UniqueName: \"kubernetes.io/projected/cacb0b7a-9b96-4964-b8cc-cab7a5c17e43-kube-api-access-xbpb2\") pod \"authorino-797fb9856f-8rq9w\" (UID: \"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43\") " pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.180212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.180192 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:42:06.213214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.213180 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797fb9856f-8rq9w" Oct 30 00:42:06.293370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.293340 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jrvcb" Oct 30 00:42:06.442391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.442309 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:06.505747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.505720 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797fb9856f-8rq9w"] Oct 30 00:42:06.509027 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:42:06.508989 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcacb0b7a_9b96_4964_b8cc_cab7a5c17e43.slice/crio-ac65ef12ccc6109fb969a3c7d1797f9f2f870f2672a6b5b5cd2e48afa5acca9a WatchSource:0}: Error finding container ac65ef12ccc6109fb969a3c7d1797f9f2f870f2672a6b5b5cd2e48afa5acca9a: Status 404 returned error can't find the container with id ac65ef12ccc6109fb969a3c7d1797f9f2f870f2672a6b5b5cd2e48afa5acca9a Oct 30 00:42:06.581748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:06.581722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:07.289112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.288540 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797fb9856f-8rq9w" event={"ID":"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43","Type":"ContainerStarted","Data":"5859c7ac629289b239912c664cb022d117ef4637716a60c35826d52d3adf6ae2"} Oct 30 00:42:07.289112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.288590 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797fb9856f-8rq9w" event={"ID":"cacb0b7a-9b96-4964-b8cc-cab7a5c17e43","Type":"ContainerStarted","Data":"ac65ef12ccc6109fb969a3c7d1797f9f2f870f2672a6b5b5cd2e48afa5acca9a"} Oct 30 00:42:07.360961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.360899 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-797fb9856f-8rq9w" podStartSLOduration=1.942527454 podStartE2EDuration="2.360858109s" totalImagesPullingTime="418.330655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.330538652 +0000 UTC m=+579.570773768" watchObservedRunningTime="2026-10-30 00:42:07.360858109 +0000 UTC m=+579.601093223" Oct 30 00:42:07.362829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.362773 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:42:07.365094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.364623 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6558f48958-22jmp" podUID="46073246-fb65-46cd-9648-ffc11a6cb904" containerName="authorino" containerID="cri-o://8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19" gracePeriod=30 Oct 30 00:42:07.632452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.632419 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:42:07.655798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.655763 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:42:07.675254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.674348 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:42:07.767549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.767514 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\" (UniqueName: \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\") pod \"46073246-fb65-46cd-9648-ffc11a6cb904\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " Oct 30 00:42:07.767727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.767553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\" (UniqueName: \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\") pod \"46073246-fb65-46cd-9648-ffc11a6cb904\" (UID: \"46073246-fb65-46cd-9648-ffc11a6cb904\") " Oct 30 00:42:07.769676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.769642 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws" pod "46073246-fb65-46cd-9648-ffc11a6cb904" (UID: "46073246-fb65-46cd-9648-ffc11a6cb904"). InnerVolumeSpecName "kube-api-access-6dgws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:42:07.777245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.777220 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert" pod "46073246-fb65-46cd-9648-ffc11a6cb904" (UID: "46073246-fb65-46cd-9648-ffc11a6cb904"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:42:07.869305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.869271 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgws\" (UniqueName: \"kubernetes.io/projected/46073246-fb65-46cd-9648-ffc11a6cb904-kube-api-access-6dgws\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:42:07.869305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:07.869301 5938 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46073246-fb65-46cd-9648-ffc11a6cb904-tls-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:42:08.294749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.294713 5938 generic.go:356] "Generic (PLEG): container finished" podID="46073246-fb65-46cd-9648-ffc11a6cb904" containerID="8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19" exitCode=0 Oct 30 00:42:08.294934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.294761 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6558f48958-22jmp" Oct 30 00:42:08.294934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.294793 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6558f48958-22jmp" event={"ID":"46073246-fb65-46cd-9648-ffc11a6cb904","Type":"ContainerDied","Data":"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19"} Oct 30 00:42:08.294934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.294836 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6558f48958-22jmp" event={"ID":"46073246-fb65-46cd-9648-ffc11a6cb904","Type":"ContainerDied","Data":"97a0cd4dadbb3ee598e907a4efcea3ffea9d3cb9cf5df1679ed33ae4304e6fdd"} Oct 30 00:42:08.294934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.294859 5938 scope.go:122] "RemoveContainer" containerID="8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19" Oct 30 00:42:08.308896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.308850 5938 scope.go:122] "RemoveContainer" containerID="8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19" Oct 30 00:42:08.309115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.309059 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" Oct 30 00:42:08.309507 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:42:08.309484 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19\": container with ID starting with 8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19 not found: ID does not exist" containerID="8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19" Oct 30 00:42:08.309582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.309517 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19"} err="failed to get container status \"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19\": rpc error: code = NotFound desc = could not find container \"8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19\": container with ID starting with 8ad50aa6c5b2d0514e055d3e589ed629da2fedd1cbe8b908f47f0e439daf1b19 not found: ID does not exist" Oct 30 00:42:08.392733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.392704 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:42:08.397041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:08.397013 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6558f48958-22jmp"] Oct 30 00:42:09.874922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:09.874884 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46073246-fb65-46cd-9648-ffc11a6cb904" path="/var/lib/kubelet/pods/46073246-fb65-46cd-9648-ffc11a6cb904/volumes" Oct 30 00:42:11.600673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.600642 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:11.738882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.738834 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv"] Oct 30 00:42:11.739278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.739263 5938 cpu_manager.go:414] "containerMap: removing container" podUID="46073246-fb65-46cd-9648-ffc11a6cb904" containerName="authorino" Oct 30 00:42:11.739328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.739278 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46073246-fb65-46cd-9648-ffc11a6cb904" containerName="authorino" Oct 30 00:42:11.739374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.739338 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="46073246-fb65-46cd-9648-ffc11a6cb904" containerName="authorino" Oct 30 00:42:11.742249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.742231 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:11.751816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.751790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:11.760317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.760294 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv"] Oct 30 00:42:11.792978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.792959 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 00:42:11.792978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.792962 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9hmph\"" Oct 30 00:42:11.905654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.905628 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cce19f-6ecc-4cf9-8205-628cf08ec662-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:11.905758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:11.905658 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxw2\" (UniqueName: \"kubernetes.io/projected/97cce19f-6ecc-4cf9-8205-628cf08ec662-kube-api-access-kgxw2\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.006906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.006857 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cce19f-6ecc-4cf9-8205-628cf08ec662-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.006998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.006916 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxw2\" (UniqueName: \"kubernetes.io/projected/97cce19f-6ecc-4cf9-8205-628cf08ec662-kube-api-access-kgxw2\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.007562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.007543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97cce19f-6ecc-4cf9-8205-628cf08ec662-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.029579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.029548 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxw2\" (UniqueName: \"kubernetes.io/projected/97cce19f-6ecc-4cf9-8205-628cf08ec662-kube-api-access-kgxw2\") pod \"payload-pre-processing-7b98bb678c-qcfkv\" (UID: \"97cce19f-6ecc-4cf9-8205-628cf08ec662\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.055380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.055356 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:12.346387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:12.346318 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv"] Oct 30 00:42:12.349474 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:42:12.349449 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cce19f_6ecc_4cf9_8205_628cf08ec662.slice/crio-22083c2cb87f37c03c01510db029732e26c27e1152f8e3c6a2e3e91af2d85d24 WatchSource:0}: Error finding container 22083c2cb87f37c03c01510db029732e26c27e1152f8e3c6a2e3e91af2d85d24: Status 404 returned error can't find the container with id 22083c2cb87f37c03c01510db029732e26c27e1152f8e3c6a2e3e91af2d85d24 Oct 30 00:42:13.319926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:13.319886 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" event={"ID":"97cce19f-6ecc-4cf9-8205-628cf08ec662","Type":"ContainerStarted","Data":"22083c2cb87f37c03c01510db029732e26c27e1152f8e3c6a2e3e91af2d85d24"} Oct 30 00:42:14.244789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.244753 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:42:14.248187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.248169 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.262843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.262807 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:42:14.262937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.262914 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwncz\"" Oct 30 00:42:14.276402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.276379 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:42:14.325833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.325807 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" event={"ID":"97cce19f-6ecc-4cf9-8205-628cf08ec662","Type":"ContainerStarted","Data":"f9cbb9c8233040c1a692696621fd0d6b279375576c8d8e2c28630d0a3079a7ce"} Oct 30 00:42:14.326235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.325931 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:14.367310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.367267 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" podStartSLOduration=1.4692257739999999 podStartE2EDuration="3.367254258s" totalImagesPullingTime="1.898028484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:14.366594742 +0000 UTC m=+586.606829854" watchObservedRunningTime="2026-10-30 00:42:14.367254258 +0000 UTC m=+586.607489371" Oct 30 00:42:14.428933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.428904 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.428938 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429051 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429096 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429255 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5ln\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429312 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429389 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429426 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.429519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.429487 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530333 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530350 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530369 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530408 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530432 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530465 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5ln\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530530 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530792 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530579 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530787 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.530857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.530808 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.531112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.531090 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.531190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.531170 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.531232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.531204 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.532834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.532813 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.532962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.532944 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.552655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.552636 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.552900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.552883 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5ln\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lxc86\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.563712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.563694 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:14.887123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:14.887093 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:42:14.890341 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:42:14.890316 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2632b1f_0282_4768_87f5_3b476146a536.slice/crio-df3d7094db9ec5f73a71ca0a5187a3bead90d38203015d6f4b4b6d26516ed4d0 WatchSource:0}: Error finding container df3d7094db9ec5f73a71ca0a5187a3bead90d38203015d6f4b4b6d26516ed4d0: Status 404 returned error can't find the container with id df3d7094db9ec5f73a71ca0a5187a3bead90d38203015d6f4b4b6d26516ed4d0 Oct 30 00:42:15.331698 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:15.331625 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" event={"ID":"e2632b1f-0282-4768-87f5-3b476146a536","Type":"ContainerStarted","Data":"df3d7094db9ec5f73a71ca0a5187a3bead90d38203015d6f4b4b6d26516ed4d0"} Oct 30 00:42:17.804252 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:17.804215 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:17.804518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:17.804287 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:17.804518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:17.804341 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:18.345963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:18.345928 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" event={"ID":"e2632b1f-0282-4768-87f5-3b476146a536","Type":"ContainerStarted","Data":"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f"} Oct 30 00:42:18.564646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:18.564606 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:18.566101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:18.566065 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:18.566231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:18.566133 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:19.564938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:19.564905 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:19.565291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:19.564959 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:20.564517 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:20.564479 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:20.564711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:20.564547 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:21.565037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:21.564997 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:21.565435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:21.565066 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:22.564500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:22.564464 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:22.564806 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:22.564527 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:23.564296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:23.564266 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:23.564667 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:23.564315 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:24.565095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:24.565058 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lxc86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 00:42:24.565498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:24.565114 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 00:42:25.264352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:25.264311 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:25.335694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:25.335664 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qcfkv" Oct 30 00:42:25.376776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:25.376729 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podStartSLOduration=8.465641389 podStartE2EDuration="11.376715556s" totalImagesPullingTime="2.911074167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:18.404384073 +0000 UTC m=+590.644619188" watchObservedRunningTime="2026-10-30 00:42:25.376715556 +0000 UTC m=+597.616950669" Oct 30 00:42:25.410140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:25.410114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:26.027191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:26.027162 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:26.027601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:26.027576 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:26.028341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:26.028318 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:42:27.936068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:27.936045 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:42:27.937148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:27.937124 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:42:28.835816 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:42:28.835768 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:42:29.291134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.291103 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:42:29.297549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.297527 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.327361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.327336 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:42:29.378437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378413 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378439 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378462 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378481 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378595 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378616 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378675 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378734 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.378802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.378770 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q26\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479177 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479228 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479246 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479276 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q26\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479314 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479328 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479383 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479440 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479598 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479661 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479836 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.479967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.479941 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.480302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.480275 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.482268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.482241 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.482423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.482406 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.506142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.502232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.506142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.502481 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q26\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ws84g\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.611103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.611079 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:29.924342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.924266 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:42:29.927129 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:42:29.927098 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94636f9e_16ba_4105_910f_624db998a7ca.slice/crio-028d807ab238d202195a9cb2ff415df4512b72d95301394da90b72c4cda9b5a1 WatchSource:0}: Error finding container 028d807ab238d202195a9cb2ff415df4512b72d95301394da90b72c4cda9b5a1: Status 404 returned error can't find the container with id 028d807ab238d202195a9cb2ff415df4512b72d95301394da90b72c4cda9b5a1 Oct 30 00:42:29.929325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.929286 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:29.929414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.929357 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:29.929414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:29.929389 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:42:30.403008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.401254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" event={"ID":"94636f9e-16ba-4105-910f-624db998a7ca","Type":"ContainerStarted","Data":"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4"} Oct 30 00:42:30.403008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.401332 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" event={"ID":"94636f9e-16ba-4105-910f-624db998a7ca","Type":"ContainerStarted","Data":"028d807ab238d202195a9cb2ff415df4512b72d95301394da90b72c4cda9b5a1"} Oct 30 00:42:30.446625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.446570 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podStartSLOduration=1.4465538709999999 podStartE2EDuration="1.446553871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.445252201 +0000 UTC m=+602.685487314" watchObservedRunningTime="2026-10-30 00:42:30.446553871 +0000 UTC m=+602.686788983" Oct 30 00:42:30.612241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.612196 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:30.613706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.613679 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:30.613782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:30.613736 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:31.611879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:31.611829 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:31.612348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:31.611904 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:32.612368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:32.612335 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:32.612761 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:32.612391 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:33.611666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:33.611628 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:33.611847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:33.611684 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:34.611796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:34.611749 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:34.612293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:34.611809 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:35.145830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:35.145797 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.602949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:35.596835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.612130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:35.612095 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:35.612475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:35.612150 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:36.121187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:36.121155 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:36.612360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:36.612330 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ws84g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 00:42:36.612733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:36.612384 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 00:42:37.645561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:37.645526 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.795169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:37.795140 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:38.093640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:38.093555 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:38.093892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:38.093852 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:38.094722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:38.094698 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:42:39.106927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:39.106841 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.252582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:39.252552 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:53.564717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:42:53.564677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:58.834463 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:42:58.834415 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:43:03.955534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:03.955477 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.092429 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.092401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.325144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.325073 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 00:43:04.333247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.333223 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.351242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.351173 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cefb27-openshift-default-dockercfg-jwt4c\"" Oct 30 00:43:04.364006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.363977 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 00:43:04.430776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.430739 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.450544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.450520 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 00:43:04.452979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.452956 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.475822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.475797 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f4fba-openshift-default-dockercfg-l52jt\"" Oct 30 00:43:04.478249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478227 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478285 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478352 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478382 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478409 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478541 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478577 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478605 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.478730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.478686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4g7\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.481800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.481777 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 00:43:04.558496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.558468 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:43:04.560971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.560935 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.574955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.574931 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29be73-openshift-default-dockercfg-xqn9v\"" Oct 30 00:43:04.576846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.576786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.579977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.579953 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.579983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580004 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580056 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580102 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580200 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580246 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580298 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580317 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580369 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580390 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580422 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgf5s\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580449 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.580635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580479 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4g7\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580565 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580565 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580613 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580637 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580654 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580661 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.580921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.580842 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.581205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.581175 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.584066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.584037 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.584687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.584667 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:43:04.584998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.584973 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.605612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.605592 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.609512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.609491 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4g7\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.646473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.646445 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:04.682055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682020 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682077 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682148 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682258 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682310 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682380 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682418 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588tf\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682455 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682572 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682680 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682730 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.682782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682790 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682852 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682894 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.682930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682914 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682948 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.682981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.683007 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.683038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgf5s\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.683062 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.683139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.683080 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.683461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.683442 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.684363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.684340 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.685365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.685300 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.699452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.697778 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:43:04.703389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.703372 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.713994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.713239 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.719890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.719843 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgf5s\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.730875 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.730844 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ca5363-openshift-default-dockercfg-m4tkh\"" Oct 30 00:43:04.739053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.739003 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:43:04.764629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.764610 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:04.783949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.783921 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.783963 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784095 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784204 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784241 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784314 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784354 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784358 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784434 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784454 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588tf\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784506 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784545 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784608 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784615 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.784955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784726 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784763 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.784955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.784849 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.785122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.785095 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.785346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.785168 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.785346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.785181 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96r6\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.785346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.785299 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.785527 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.785509 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.787332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.787306 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.787477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.787457 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.822471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.822333 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.825359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.825328 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-588tf\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.851784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.851707 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.874644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.874613 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:04.883778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.883754 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:43:04.886455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886437 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.886680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96r6\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886663 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886694 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886740 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.886894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886837 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886895 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.886931 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.887186 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.887390 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.887536 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.887646 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.887974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.887952 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.889401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.889381 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.889688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.889672 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.903358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.903337 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-021d6d-openshift-default-dockercfg-trf72\"" Oct 30 00:43:04.913008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.911874 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:43:04.919714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.919663 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96r6\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.919841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.919718 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:04.973709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.973666 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 00:43:04.977583 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:04.977558 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e48bff3_a057_4901_bb9a_8efa404e8ae9.slice/crio-6564a9d3b2fc036a539df61cbc99a11e74d8fd07089a53e39b1806357289af1d WatchSource:0}: Error finding container 6564a9d3b2fc036a539df61cbc99a11e74d8fd07089a53e39b1806357289af1d: Status 404 returned error can't find the container with id 6564a9d3b2fc036a539df61cbc99a11e74d8fd07089a53e39b1806357289af1d Oct 30 00:43:04.979764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.979727 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:04.979854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.979796 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:04.979854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.979842 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:04.988025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988000 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988045 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988064 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzds\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988165 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988212 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988259 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988284 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988326 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:04.988485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:04.988353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.018514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.018489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:05.021106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.021085 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:05.027012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.026747 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:43:05.029785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.029767 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.054039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.054008 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:43:05.066524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.066306 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089582 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089663 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089723 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089754 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzds\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089796 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089849 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089917 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.089960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089957 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.090413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.089994 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.090853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.090711 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1a11cf-openshift-default-dockercfg-j69mc\"" Oct 30 00:43:05.091705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.091638 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.092407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.092110 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.092407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.092327 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.092562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.092525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.094285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.092742 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.094285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.092743 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.111352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.096797 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.111352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.097574 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.111352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.097690 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.111352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.107835 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.116354 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.116305 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzds\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.116598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.116573 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.178649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.178603 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:05.190339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190313 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:43:05.190689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190663 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.190825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190711 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28bl\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.190825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190747 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.191044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190940 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.191044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.190976 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.191044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.191040 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.192474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.191093 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.192474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.191180 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.192474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.191377 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.194800 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:05.194769 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b3bb54_064e_4373_b226_08b02119b430.slice/crio-d2faf8d77f097f084bae280abc852ed7883dafb5b7c4d117a7751a29f7f35c83 WatchSource:0}: Error finding container d2faf8d77f097f084bae280abc852ed7883dafb5b7c4d117a7751a29f7f35c83: Status 404 returned error can't find the container with id d2faf8d77f097f084bae280abc852ed7883dafb5b7c4d117a7751a29f7f35c83 Oct 30 00:43:05.197143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.197116 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.197212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.197195 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.197251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.197228 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.202902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.202879 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.293979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294046 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28bl\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294084 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294116 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294150 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294227 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294288 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294354 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.294942 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.295202 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.296127 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.298296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.297613 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.300815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.300242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.306962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.306918 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.307526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.307488 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.321052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.321010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28bl\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.322209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.322180 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.326954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.326878 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:05.353101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.352778 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:05.355543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.355449 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:43:05.378965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.378786 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.378965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.378956 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.379130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.379036 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.503903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.502136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:05.531465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.531432 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:43:05.535730 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:05.535704 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b06c81b_19fa_4af1_9da1_b9a6dee53e53.slice/crio-c301990bed963999c2a0ae91cfaedd1b6511128f26642264af5575769286ee7c WatchSource:0}: Error finding container c301990bed963999c2a0ae91cfaedd1b6511128f26642264af5575769286ee7c: Status 404 returned error can't find the container with id c301990bed963999c2a0ae91cfaedd1b6511128f26642264af5575769286ee7c Oct 30 00:43:05.539241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.538823 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.539241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.538930 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.539241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.538964 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.573563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.573035 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerStarted","Data":"f095fd72cbb94f525a29e07b2e1961e90abc9c2789a7383820e6287682ca3902"} Oct 30 00:43:05.573563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.573093 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerStarted","Data":"d1b6cfffe814999d8659b8b9facd3d906193b458f2cfd06782e2a2681fe2610d"} Oct 30 00:43:05.575328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.575291 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerStarted","Data":"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600"} Oct 30 00:43:05.575441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.575336 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerStarted","Data":"d2faf8d77f097f084bae280abc852ed7883dafb5b7c4d117a7751a29f7f35c83"} Oct 30 00:43:05.577161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.577109 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerStarted","Data":"c301990bed963999c2a0ae91cfaedd1b6511128f26642264af5575769286ee7c"} Oct 30 00:43:05.579333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.578627 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerStarted","Data":"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162"} Oct 30 00:43:05.579333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.578669 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerStarted","Data":"6564a9d3b2fc036a539df61cbc99a11e74d8fd07089a53e39b1806357289af1d"} Oct 30 00:43:05.581094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.581068 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerStarted","Data":"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a"} Oct 30 00:43:05.581195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.581117 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerStarted","Data":"ab5fc90e4f1edbd64063ed7a4e4a75328796df8a1a0f86ade63e1d01f721895d"} Oct 30 00:43:05.618610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.618526 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podStartSLOduration=1.618510953 podStartE2EDuration="1.618510953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.617211026 +0000 UTC m=+637.857446140" watchObservedRunningTime="2026-10-30 00:43:05.618510953 +0000 UTC m=+637.858746067" Oct 30 00:43:05.647207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.647167 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:05.663111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.662803 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podStartSLOduration=1.662789306 podStartE2EDuration="1.662789306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.662072035 +0000 UTC m=+637.902307148" watchObservedRunningTime="2026-10-30 00:43:05.662789306 +0000 UTC m=+637.903024415" Oct 30 00:43:05.692452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.692412 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:43:05.712070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.707712 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podStartSLOduration=1.707692148 podStartE2EDuration="1.707692148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.705688978 +0000 UTC m=+637.945924093" watchObservedRunningTime="2026-10-30 00:43:05.707692148 +0000 UTC m=+637.947927262" Oct 30 00:43:05.712070 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:05.708522 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb14d4d8_31e4_4a02_925c_03fc3d350026.slice/crio-5e3c25a035c4866d24f0254c21c9eff0c74fb2d199ceae8e7ec54e496e6a2025 WatchSource:0}: Error finding container 5e3c25a035c4866d24f0254c21c9eff0c74fb2d199ceae8e7ec54e496e6a2025: Status 404 returned error can't find the container with id 5e3c25a035c4866d24f0254c21c9eff0c74fb2d199ceae8e7ec54e496e6a2025 Oct 30 00:43:05.715801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.715770 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.715927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.715840 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.715927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.715903 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:05.765287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.765266 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:05.890921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:05.888722 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:06.022071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.022036 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:06.027359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.027333 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:06.071074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.071021 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podStartSLOduration=2.07098206 podStartE2EDuration="2.07098206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.752776185 +0000 UTC m=+637.993011298" watchObservedRunningTime="2026-10-30 00:43:06.07098206 +0000 UTC m=+638.311217175" Oct 30 00:43:06.587667 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.587630 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerStarted","Data":"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba"} Oct 30 00:43:06.587856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.587693 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerStarted","Data":"5e3c25a035c4866d24f0254c21c9eff0c74fb2d199ceae8e7ec54e496e6a2025"} Oct 30 00:43:06.589573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.589543 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerStarted","Data":"6a895a218bd28f0969b39923e0844de7bb22d8f2d319a1c85eda3e288124b62a"} Oct 30 00:43:06.589706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.589689 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:06.590626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.590605 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:06.631532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.631485 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podStartSLOduration=2.631469664 podStartE2EDuration="2.631469664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:06.631070103 +0000 UTC m=+638.871305217" watchObservedRunningTime="2026-10-30 00:43:06.631469664 +0000 UTC m=+638.871704777" Oct 30 00:43:06.651725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.651699 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:06.676903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.676840 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podStartSLOduration=2.676827222 podStartE2EDuration="2.676827222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:06.675204626 +0000 UTC m=+638.915439775" watchObservedRunningTime="2026-10-30 00:43:06.676827222 +0000 UTC m=+638.917062390" Oct 30 00:43:06.770242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.770209 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:06.880377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:06.880349 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:07.203742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.203671 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:07.208518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.208495 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:07.353992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.353961 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:07.358909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.358883 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:07.596392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596124 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:07.596597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596574 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:07.596721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596609 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:07.596721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596628 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:07.596721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596643 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:07.596721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596684 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:07.596721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596713 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:07.596928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596817 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:07.596928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596845 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:07.596928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:07.596894 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:08.203606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:08.203580 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.374666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:08.374631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.515534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:08.515467 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.671696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:08.671666 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.900625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:08.900587 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:09.053932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:09.053848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:09.212656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:09.212585 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:09.396281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:09.396250 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:09.542271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:09.542196 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:10.099690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:10.099661 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:10.244210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:10.244176 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.844520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:11.844481 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.983114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:11.983083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:12.139726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:12.138348 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:12.832676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:12.832644 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:12.974819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:12.974787 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:13.784711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:13.784682 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:13.964706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:13.964677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:15.729999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:15.729962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:16.445126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.445087 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 00:43:16.449072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.449049 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.466830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.466810 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 00:43:16.476280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.476258 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-52650f\"" Oct 30 00:43:16.476280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.476257 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-52650f-dockercfg-nw9dj\"" Oct 30 00:43:16.554818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.554784 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 00:43:16.559848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.559200 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.576056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.576033 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 00:43:16.645382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.645349 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4658\" (UniqueName: \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.645541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.645481 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.746792 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.746664 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.747297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.746899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4658\" (UniqueName: \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.747297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.746960 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn6h\" (UniqueName: \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.747297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.747040 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.747567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.747535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.769364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.769331 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4658\" (UniqueName: \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\") pod \"payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:16.847597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.847562 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn6h\" (UniqueName: \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.847741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.847647 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.848369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.848349 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.870077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.870048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn6h\" (UniqueName: \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\") pod \"payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.872811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.872790 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:16.930308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:16.930251 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:17.061359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.061277 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:17.214681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.214653 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 00:43:17.216617 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:17.216589 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7482d7ef_6951_40d2_b581_39b19c0d35a2.slice/crio-5f612468b0a16eeb947243cb89c4e27866c988bdac35220cfd5a4997d4f80f53 WatchSource:0}: Error finding container 5f612468b0a16eeb947243cb89c4e27866c988bdac35220cfd5a4997d4f80f53: Status 404 returned error can't find the container with id 5f612468b0a16eeb947243cb89c4e27866c988bdac35220cfd5a4997d4f80f53 Oct 30 00:43:17.317922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.317813 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:17.387886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.384209 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 00:43:17.401150 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:17.400367 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489d8d84_065a_455b_aba1_298d85025ca7.slice/crio-46f50304e666d4264931c764f70d52677a4ed9db2e1612943863ffb68155d494 WatchSource:0}: Error finding container 46f50304e666d4264931c764f70d52677a4ed9db2e1612943863ffb68155d494: Status 404 returned error can't find the container with id 46f50304e666d4264931c764f70d52677a4ed9db2e1612943863ffb68155d494 Oct 30 00:43:17.472606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.472564 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:17.644407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.644366 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" event={"ID":"489d8d84-065a-455b-aba1-298d85025ca7","Type":"ContainerStarted","Data":"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87"} Oct 30 00:43:17.644588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.644439 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:17.644588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.644459 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" event={"ID":"489d8d84-065a-455b-aba1-298d85025ca7","Type":"ContainerStarted","Data":"46f50304e666d4264931c764f70d52677a4ed9db2e1612943863ffb68155d494"} Oct 30 00:43:17.646004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.645977 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" event={"ID":"7482d7ef-6951-40d2-b581-39b19c0d35a2","Type":"ContainerStarted","Data":"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e"} Oct 30 00:43:17.646108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.646023 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" event={"ID":"7482d7ef-6951-40d2-b581-39b19c0d35a2","Type":"ContainerStarted","Data":"5f612468b0a16eeb947243cb89c4e27866c988bdac35220cfd5a4997d4f80f53"} Oct 30 00:43:17.646151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.646124 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:17.687885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.684809 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" podStartSLOduration=1.684790362 podStartE2EDuration="1.684790362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:17.682721066 +0000 UTC m=+649.922956179" watchObservedRunningTime="2026-10-30 00:43:17.684790362 +0000 UTC m=+649.925025477" Oct 30 00:43:17.732261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:17.732217 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" podStartSLOduration=1.7322040520000002 podStartE2EDuration="1.732204052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:17.730666154 +0000 UTC m=+649.970901267" watchObservedRunningTime="2026-10-30 00:43:17.732204052 +0000 UTC m=+649.972439194" Oct 30 00:43:18.041025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.040949 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:18.152192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.152158 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 00:43:18.156727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.156700 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.160990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.160959 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8tbl\" (UniqueName: \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.161133 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.161052 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.174422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.174396 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-021d6d-dockercfg-c5b6g\"" Oct 30 00:43:18.174615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.174427 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-021d6d\"" Oct 30 00:43:18.175047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.175019 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 00:43:18.216648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.216607 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:18.245221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.244256 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 00:43:18.261884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.257114 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.319944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.279906 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnv5\" (UniqueName: \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.321144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.321048 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8tbl\" (UniqueName: \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.321144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.321075 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 00:43:18.321144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.321150 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.323056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.321243 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.323056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.322645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.350741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.350717 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8tbl\" (UniqueName: \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\") pod \"payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.422516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.422472 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.422697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.422566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnv5\" (UniqueName: \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.423335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.423310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.447663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.447634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnv5\" (UniqueName: \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\") pod \"payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.472571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.472535 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:18.644172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.644142 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:18.870823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.870793 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 00:43:18.873265 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:18.873238 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6e60b5_afa2_4250_b14a_f4d97f3315f3.slice/crio-f6315229ad1934f0bf09186f5d2d210bc11c5e5b2a9309d7875cbca62020827c WatchSource:0}: Error finding container f6315229ad1934f0bf09186f5d2d210bc11c5e5b2a9309d7875cbca62020827c: Status 404 returned error can't find the container with id f6315229ad1934f0bf09186f5d2d210bc11c5e5b2a9309d7875cbca62020827c Oct 30 00:43:18.916795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.916766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:18.979957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:18.978594 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 00:43:18.981796 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:18.981767 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode727a883_9c06_4c7e_a136_605055f927ae.slice/crio-3056becaec0b6d04207a589c3c0f5056180cba3a55010e24720bc7a7c3d888cb WatchSource:0}: Error finding container 3056becaec0b6d04207a589c3c0f5056180cba3a55010e24720bc7a7c3d888cb: Status 404 returned error can't find the container with id 3056becaec0b6d04207a589c3c0f5056180cba3a55010e24720bc7a7c3d888cb Oct 30 00:43:19.066909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.066851 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.172309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.163982 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 00:43:19.233281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.233244 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 00:43:19.233459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.233361 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.245508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.245476 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.245772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.245752 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjn75\" (UniqueName: \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.250975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.250292 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-1a11cf\"" Oct 30 00:43:19.254721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.254603 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.297891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.280199 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-1a11cf-dockercfg-vj2sk\"" Oct 30 00:43:19.354295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.354252 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.357192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.357162 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjn75\" (UniqueName: \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.357724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.356648 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.390889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.389552 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjn75\" (UniqueName: \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\") pod \"payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.413890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.410602 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.558982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.558888 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.561285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.561122 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:19.676704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.675756 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" event={"ID":"5f6e60b5-afa2-4250-b14a-f4d97f3315f3","Type":"ContainerStarted","Data":"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1"} Oct 30 00:43:19.676704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.675831 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" event={"ID":"5f6e60b5-afa2-4250-b14a-f4d97f3315f3","Type":"ContainerStarted","Data":"f6315229ad1934f0bf09186f5d2d210bc11c5e5b2a9309d7875cbca62020827c"} Oct 30 00:43:19.676704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.676660 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:19.678738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.678256 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" event={"ID":"e727a883-9c06-4c7e-a136-605055f927ae","Type":"ContainerStarted","Data":"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4"} Oct 30 00:43:19.678738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.678292 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" event={"ID":"e727a883-9c06-4c7e-a136-605055f927ae","Type":"ContainerStarted","Data":"3056becaec0b6d04207a589c3c0f5056180cba3a55010e24720bc7a7c3d888cb"} Oct 30 00:43:19.678738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.678697 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:19.768973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.768945 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.815942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.804201 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" podStartSLOduration=1.804180092 podStartE2EDuration="1.804180092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:19.803346981 +0000 UTC m=+652.043582096" watchObservedRunningTime="2026-10-30 00:43:19.804180092 +0000 UTC m=+652.044415207" Oct 30 00:43:19.842923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.833272 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" podStartSLOduration=1.8332541519999999 podStartE2EDuration="1.833254152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:19.733716739 +0000 UTC m=+651.973951853" watchObservedRunningTime="2026-10-30 00:43:19.833254152 +0000 UTC m=+652.073489267" Oct 30 00:43:19.971153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:19.971025 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 00:43:20.004655 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:20.004518 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67c1fd_69f6_41a5_a599_277101dfd095.slice/crio-1a6fa75823295e87f3b26acd13d9f7e2a1362210b811d88aeea3e3eedfe8fdde WatchSource:0}: Error finding container 1a6fa75823295e87f3b26acd13d9f7e2a1362210b811d88aeea3e3eedfe8fdde: Status 404 returned error can't find the container with id 1a6fa75823295e87f3b26acd13d9f7e2a1362210b811d88aeea3e3eedfe8fdde Oct 30 00:43:20.693889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:20.689610 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" event={"ID":"fe67c1fd-69f6-41a5-a599-277101dfd095","Type":"ContainerStarted","Data":"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157"} Oct 30 00:43:20.693889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:20.689672 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" event={"ID":"fe67c1fd-69f6-41a5-a599-277101dfd095","Type":"ContainerStarted","Data":"1a6fa75823295e87f3b26acd13d9f7e2a1362210b811d88aeea3e3eedfe8fdde"} Oct 30 00:43:20.693889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:20.690192 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:20.732215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:20.732154 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" podStartSLOduration=1.732134968 podStartE2EDuration="1.732134968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:20.730197578 +0000 UTC m=+652.970432692" watchObservedRunningTime="2026-10-30 00:43:20.732134968 +0000 UTC m=+652.972370082" Oct 30 00:43:20.951976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:20.951852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:22.649397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:22.649364 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:23.056630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.056545 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:23.646060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.646017 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:43:23.646252 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.646082 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:43:23.648603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.648568 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:23.814561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.814425 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:23.994489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:23.994341 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:24.775957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:24.775407 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:24.923485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:24.923415 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.270393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.270351 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.759739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.759660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.815842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.815790 5938 generic.go:356] "Generic (PLEG): container finished" podID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerID="ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162" exitCode=0 Oct 30 00:43:26.816055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.815905 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerDied","Data":"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162"} Oct 30 00:43:26.816442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.816423 5938 scope.go:122] "RemoveContainer" containerID="ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162" Oct 30 00:43:26.819048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.818046 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:26.819048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.818133 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:26.819048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.818176 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:26.911907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:26.906200 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:27.840891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:27.839347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerStarted","Data":"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b"} Oct 30 00:43:28.245899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.232515 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:28.441674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.441634 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:28.646819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.646761 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:28.647061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.646918 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:43:28.647061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.646966 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:43:28.664803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.664777 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 00:43:28.665745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.665708 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 00:43:28.783724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.783672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:28.874594 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:43:28.874555 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:43:28.965815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:28.965658 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:29.119813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:29.119708 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:29.647848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:29.647460 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:43:29.647848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:29.647542 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:43:30.342910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:30.342820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:30.647605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:30.647471 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:43:30.647605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:30.647536 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:43:31.649923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:31.649352 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:43:31.649923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:31.649411 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:43:31.741891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:31.708724 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 00:43:31.741891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:31.718796 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 00:43:32.185800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:32.181191 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:32.658457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:32.648970 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 00:43:32.658457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:32.649035 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 00:43:32.740899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:32.739884 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 00:43:33.381647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:33.381489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:33.861091 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:33.861016 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:34.006385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.005051 5938 generic.go:356] "Generic (PLEG): container finished" podID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerID="29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a" exitCode=0 Oct 30 00:43:34.006385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.005177 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerDied","Data":"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a"} Oct 30 00:43:34.006385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.005645 5938 scope.go:122] "RemoveContainer" containerID="29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a" Oct 30 00:43:34.013158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.013108 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.013412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.013386 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.013605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.013567 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.032213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.031162 5938 generic.go:356] "Generic (PLEG): container finished" podID="a571529c-e955-4307-9501-7c97b832cff8" containerID="f095fd72cbb94f525a29e07b2e1961e90abc9c2789a7383820e6287682ca3902" exitCode=0 Oct 30 00:43:34.032213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.031205 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerDied","Data":"f095fd72cbb94f525a29e07b2e1961e90abc9c2789a7383820e6287682ca3902"} Oct 30 00:43:34.032213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.031575 5938 scope.go:122] "RemoveContainer" containerID="f095fd72cbb94f525a29e07b2e1961e90abc9c2789a7383820e6287682ca3902" Oct 30 00:43:34.033930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.033602 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.033930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.033681 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.033930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.033728 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:34.069907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.054334 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:34.251577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.251384 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:34.273045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.272138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:34.274007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.273562 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:34.274886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:34.274765 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 00:43:35.060608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.060549 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerStarted","Data":"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532"} Oct 30 00:43:35.065962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.065921 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerStarted","Data":"cccb9561a44836783253d1f7d7059844dea99c7a0cf9e86fa67ba829812e772b"} Oct 30 00:43:35.350063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.347082 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:35.769404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.769370 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:35.769778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.769757 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:35.769965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:35.769941 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:36.023962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.023570 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:43:36.023962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.023635 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 00:43:36.024213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.023973 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:36.106017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.099025 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:36.594419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.594339 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:36.765745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.765707 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:36.765950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:36.765775 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:37.027906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:37.024251 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:43:37.027906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:37.024312 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 00:43:37.605933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:37.597370 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:37.771909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:37.771312 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:37.771909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:37.771379 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:38.027749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.027256 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:43:38.027749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.027317 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 00:43:38.099020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.097749 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:38.513551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.513498 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:38.639308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.639000 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:43:38.639308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.639065 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:43:38.643167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.643018 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:43:38.643167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.643096 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:43:38.643704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.643122 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:43:38.643704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.643405 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:43:38.766139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.766024 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:38.766139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.766079 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:38.811047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:38.810948 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.022807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.022292 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:43:39.022807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.022351 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 00:43:39.094902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.094023 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.280641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.280513 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.463914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.463854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.647615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.647161 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.766208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.765885 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:39.766208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:39.765949 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:40.003251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:40.003129 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:40.025648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:40.025227 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:43:40.025648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:40.025285 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 00:43:41.056413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:40.765562 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 00:43:41.056413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:40.765620 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 00:43:41.135293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.135254 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:41.139058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.138634 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:43:41.161709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.161677 5938 generic.go:356] "Generic (PLEG): container finished" podID="b3b3bb54-064e-4373-b226-08b02119b430" containerID="4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600" exitCode=0 Oct 30 00:43:41.161904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.161757 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerDied","Data":"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600"} Oct 30 00:43:41.162341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.162323 5938 scope.go:122] "RemoveContainer" containerID="4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600" Oct 30 00:43:41.163738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.163713 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:41.163808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.163789 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:41.163857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.163822 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:41.332390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.332290 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:41.770244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.770211 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:41.771325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.771300 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 00:43:41.828904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:41.828811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:42.170599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:42.170562 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerStarted","Data":"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98"} Oct 30 00:43:42.875084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:42.875054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:42.875299 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:42.875233 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 00:43:42.875376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:42.875300 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 00:43:43.178406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.178376 5938 generic.go:356] "Generic (PLEG): container finished" podID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerID="b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba" exitCode=0 Oct 30 00:43:43.178847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.178451 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerDied","Data":"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba"} Oct 30 00:43:43.178847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.178835 5938 scope.go:122] "RemoveContainer" containerID="b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba" Oct 30 00:43:43.180165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180135 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.180267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180212 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.180267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180253 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.180355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180323 5938 generic.go:356] "Generic (PLEG): container finished" podID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerID="6a895a218bd28f0969b39923e0844de7bb22d8f2d319a1c85eda3e288124b62a" exitCode=0 Oct 30 00:43:43.180423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180401 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerDied","Data":"6a895a218bd28f0969b39923e0844de7bb22d8f2d319a1c85eda3e288124b62a"} Oct 30 00:43:43.180977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.180961 5938 scope.go:122] "RemoveContainer" containerID="6a895a218bd28f0969b39923e0844de7bb22d8f2d319a1c85eda3e288124b62a" Oct 30 00:43:43.182132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.182034 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.182132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.182090 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.182132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.182126 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:43:43.716744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.716700 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:43.874984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.874950 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 00:43:43.875184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:43.875013 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 00:43:44.211895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.194520 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerStarted","Data":"97c206a165436ec46b8953ff403f35baa0071b422b17a95a2782a283574e06d9"} Oct 30 00:43:44.211895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.206030 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:44.211895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.206455 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:44.211895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.206585 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:44.211895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.209098 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerStarted","Data":"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a"} Oct 30 00:43:44.356312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.356226 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:44.356601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.356577 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:44.356796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.356768 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:44.565501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.565407 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 00:43:44.569759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.569730 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.593203 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"openshift-service-ca.crt\"" Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.596051 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.596235 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"default-dockercfg-lm88z\"" Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.596353 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"simulator-w4-ca5363-kserve-self-signed-certs\"" Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.599989 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"kube-root-ca.crt\"" Oct 30 00:43:44.615888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.613121 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.670839 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.670962 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.671033 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.671068 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rbb\" (UniqueName: \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.671130 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.671332 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.671167 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.768606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.768570 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:44.772326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rbb\" (UniqueName: \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772422 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772463 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.772807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772790 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.773008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.772979 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.773423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.773397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.774959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.774935 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.776899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.776857 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.820929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.820544 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rbb\" (UniqueName: \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\") pod \"simulator-w4-ca5363-kserve-6547b69dc8-pd6v7\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:44.875313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.875267 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 00:43:44.877230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.877198 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 00:43:44.899645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:44.899611 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:45.175200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.175167 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:45.257883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.239418 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:45.257883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.252999 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:45.372886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.362217 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:45.372886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.362277 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:45.515934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.477642 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:45.867953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.710938 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 00:43:45.898913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.874977 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:45.899213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.880273 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 00:43:45.899424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:45.899397 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 00:43:46.213988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.213953 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:46.214219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.214191 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:46.374951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.355265 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:46.374951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.355346 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:46.414009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.413970 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" event={"ID":"f984a109-45ff-4560-8ed4-47457f5b29e6","Type":"ContainerStarted","Data":"0c24499999862ea32c82cc698b015ede02b2fef32a7bc24698090336b12a6dda"} Oct 30 00:43:46.875456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.875425 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 00:43:46.875686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:46.875663 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 00:43:47.206381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.206288 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:47.206650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.206621 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:47.355885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.355588 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:47.355885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.355647 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:47.430777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.430739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" event={"ID":"f984a109-45ff-4560-8ed4-47457f5b29e6","Type":"ContainerStarted","Data":"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c"} Oct 30 00:43:47.527940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.527740 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podStartSLOduration=3.527718024 podStartE2EDuration="3.527718024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:47.519252537 +0000 UTC m=+679.759487650" watchObservedRunningTime="2026-10-30 00:43:47.527718024 +0000 UTC m=+679.767953138" Oct 30 00:43:47.609717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.609664 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:47.883104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.883071 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:47.883570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.883542 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:47.896139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.896116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:43:47.900838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:47.900811 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:48.205944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:48.203513 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:48.205944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:48.203579 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:48.353611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:48.353578 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:48.353888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:48.353842 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:49.214942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:49.207277 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:43:49.214942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:49.207340 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 00:43:49.353570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:49.353543 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:43:49.353807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:49.353785 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 00:43:49.424893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:49.418851 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:50.208265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.208232 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:50.208629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.208610 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:50.209434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.209412 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:43:50.358543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.358507 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:50.359230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.359198 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:50.359783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:50.359764 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:43:51.179603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:51.179550 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:52.830823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:52.830781 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:52.991956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:52.991919 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:54.025574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:54.025545 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:54.579219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:54.579177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:54.900858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:54.900825 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:54.924460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:54.924427 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:55.492375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:55.492348 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:43:56.085099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:56.085056 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:57.077428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.077395 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 00:43:57.080503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.080477 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.111477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.111452 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"kube-root-ca.crt\"" Oct 30 00:43:57.111691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.111660 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"default-dockercfg-jzvk5\"" Oct 30 00:43:57.111794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.111686 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"openshift-service-ca.crt\"" Oct 30 00:43:57.111794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.111768 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"premium-w3-cefb27-kserve-self-signed-certs\"" Oct 30 00:43:57.114478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.114454 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 00:43:57.131289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131265 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.131381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131326 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.131443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131425 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.131493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131452 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.131560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131539 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.131604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.131573 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lcp\" (UniqueName: \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.199134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.199106 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:57.232054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232026 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232057 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232125 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lcp\" (UniqueName: \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232168 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232235 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232528 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.232692 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.232594 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.234237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.234214 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.234475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.234459 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.254647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.254628 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lcp\" (UniqueName: \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\") pod \"premium-w3-cefb27-kserve-5c5fd96f8-8c7qz\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.394342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.394313 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:57.692402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.692212 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 00:43:57.694962 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:43:57.694932 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5160a8f7_8d30_4938_81b6_7140cd82c4db.slice/crio-e40b000c4928386b9b91e13c44c28e105ce0699cee67e2025f7a0fbc0848d794 WatchSource:0}: Error finding container e40b000c4928386b9b91e13c44c28e105ce0699cee67e2025f7a0fbc0848d794: Status 404 returned error can't find the container with id e40b000c4928386b9b91e13c44c28e105ce0699cee67e2025f7a0fbc0848d794 Oct 30 00:43:57.866130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:57.866077 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:58.515887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:58.513773 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" event={"ID":"5160a8f7-8d30-4938-81b6-7140cd82c4db","Type":"ContainerStarted","Data":"3e63f5cae5dd687d721bb29ab119a768d91282fbf7adfffaaa1d501a06d028fa"} Oct 30 00:43:58.515887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:58.513827 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" event={"ID":"5160a8f7-8d30-4938-81b6-7140cd82c4db","Type":"ContainerStarted","Data":"e40b000c4928386b9b91e13c44c28e105ce0699cee67e2025f7a0fbc0848d794"} Oct 30 00:43:58.561129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:58.560982 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podStartSLOduration=1.560961105 podStartE2EDuration="1.560961105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:58.557535148 +0000 UTC m=+690.797770272" watchObservedRunningTime="2026-10-30 00:43:58.560961105 +0000 UTC m=+690.801196220" Oct 30 00:43:58.850936 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:43:58.849060 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:43:59.395075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:59.395034 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:43:59.583656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:43:59.583618 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:00.374520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:00.374490 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:01.259568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:01.259532 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:02.666414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.666382 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 00:44:02.669353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.669335 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.685053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.685033 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"simulator-w2-1a11cf-kserve-self-signed-certs\"" Oct 30 00:44:02.696785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.696753 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"default-dockercfg-v7tcc\"" Oct 30 00:44:02.696934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.696884 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"kube-root-ca.crt\"" Oct 30 00:44:02.696934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.696915 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"openshift-service-ca.crt\"" Oct 30 00:44:02.698750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.698730 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 00:44:02.789479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.789435 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:02.796563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796534 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.796658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796597 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.796658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796629 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.796731 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.796826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796813 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.796894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.796853 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq29j\" (UniqueName: \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898369 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898418 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898467 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898559 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898616 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq29j\" (UniqueName: \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898672 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898730 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898798 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.898969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.898952 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.900775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.900755 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.901096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.901070 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.920988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.920941 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq29j\" (UniqueName: \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\") pod \"simulator-w2-1a11cf-kserve-7f895d686-j8dnh\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:02.983211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:02.983177 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:03.020069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.019572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:03.299195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.299129 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 00:44:03.303208 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:03.303182 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod492caa93_1750_4f0a_85ad_1bcea6a4aea0.slice/crio-463894360d8ec7196804045ca86fa6b87f0cf148bbce750e0b9209a49e93b421 WatchSource:0}: Error finding container 463894360d8ec7196804045ca86fa6b87f0cf148bbce750e0b9209a49e93b421: Status 404 returned error can't find the container with id 463894360d8ec7196804045ca86fa6b87f0cf148bbce750e0b9209a49e93b421 Oct 30 00:44:03.545782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.545749 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" event={"ID":"492caa93-1750-4f0a-85ad-1bcea6a4aea0","Type":"ContainerStarted","Data":"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb"} Oct 30 00:44:03.545967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.545794 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" event={"ID":"492caa93-1750-4f0a-85ad-1bcea6a4aea0","Type":"ContainerStarted","Data":"463894360d8ec7196804045ca86fa6b87f0cf148bbce750e0b9209a49e93b421"} Oct 30 00:44:03.587359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.587278 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podStartSLOduration=1.587265935 podStartE2EDuration="1.587265935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:03.585444217 +0000 UTC m=+695.825679339" watchObservedRunningTime="2026-10-30 00:44:03.587265935 +0000 UTC m=+695.827501047" Oct 30 00:44:03.984327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:03.984297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:04.654362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:04.654318 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:06.267913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:06.267835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:07.403886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:07.397050 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:44:07.422511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:07.422476 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:44:07.581212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:07.581181 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:44:08.079473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:08.079438 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:09.592750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:09.592705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:11.240601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:11.240558 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:12.678925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.678887 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 00:44:12.681525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.681509 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.710944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.710912 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"premium-w4-ca5363-kserve-self-signed-certs\"" Oct 30 00:44:12.714018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.713990 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 00:44:12.790366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.790333 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:12.814524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814496 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.814630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.814630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814568 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.814705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814645 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.814705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814690 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.814810 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.814786 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x28q\" (UniqueName: \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.915657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.915624 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.915657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.915663 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.915960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.915685 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.915960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.915811 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.916080 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.915983 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.916140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.916082 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x28q\" (UniqueName: \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.916193 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.916154 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.916193 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.916166 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.916351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.916326 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.918087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.918064 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.918356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.918339 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.938512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.938452 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x28q\" (UniqueName: \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\") pod \"premium-w4-ca5363-kserve-7cb6dcd795-62srm\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:12.969804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.969779 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:12.986067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.983775 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:12.996887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:12.993493 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:13.003662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.003584 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:13.296381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.296355 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 00:44:13.297751 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:13.297728 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a05680_d6ad_401e_ac93_92248c1f3886.slice/crio-addd2a9c8e6466a168e2218cfe00e0be8211490915bb232235760f44f7971793 WatchSource:0}: Error finding container addd2a9c8e6466a168e2218cfe00e0be8211490915bb232235760f44f7971793: Status 404 returned error can't find the container with id addd2a9c8e6466a168e2218cfe00e0be8211490915bb232235760f44f7971793 Oct 30 00:44:13.603472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.603439 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" event={"ID":"d3a05680-d6ad-401e-ac93-92248c1f3886","Type":"ContainerStarted","Data":"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda"} Oct 30 00:44:13.603656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.603484 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" event={"ID":"d3a05680-d6ad-401e-ac93-92248c1f3886","Type":"ContainerStarted","Data":"addd2a9c8e6466a168e2218cfe00e0be8211490915bb232235760f44f7971793"} Oct 30 00:44:13.614877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.614842 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:44:13.645753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.645704 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podStartSLOduration=1.64569119 podStartE2EDuration="1.64569119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:13.644312822 +0000 UTC m=+705.884547936" watchObservedRunningTime="2026-10-30 00:44:13.64569119 +0000 UTC m=+705.885926303" Oct 30 00:44:13.996315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:13.996241 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:14.685194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:14.684982 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:16.203399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:16.203358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:17.827289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:17.827246 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:19.476604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:19.476566 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:21.131999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:21.131964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:22.404139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.404108 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 00:44:22.407569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.407549 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417064 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417210 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417258 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hz5\" (UniqueName: \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417325 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417382 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.417696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.417471 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.436455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.436414 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"default-dockercfg-ffpkz\"" Oct 30 00:44:22.436455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.436418 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"kube-root-ca.crt\"" Oct 30 00:44:22.436627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.436491 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"openshift-service-ca.crt\"" Oct 30 00:44:22.436736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.436720 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6f4fba\"/\"premium-w5-6f4fba-kserve-self-signed-certs\"" Oct 30 00:44:22.438456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.438424 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 00:44:22.518751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.518723 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.518906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.518767 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hz5\" (UniqueName: \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.518906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.518828 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.518906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.518898 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.519070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.518981 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.519070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.519031 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.519251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.519225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.519339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.519308 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.519550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.519531 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.521457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.521433 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.521526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.521502 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.526677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.526658 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:22.546004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.545977 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hz5\" (UniqueName: \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\") pod \"premium-w5-6f4fba-kserve-c9c955b96-9r7gh\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.722668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.722583 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:22.824535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.823410 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:22.995796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:22.995725 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:23.008940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.008911 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:23.034977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.034807 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 00:44:23.038666 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:23.038628 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e777165_f922_436a_a9a5_d21765c8ff20.slice/crio-1bc02b6bbd9ad932cba8ba6a1f2761c6d15ad7872ffcc22b7e0fe10769b02cd7 WatchSource:0}: Error finding container 1bc02b6bbd9ad932cba8ba6a1f2761c6d15ad7872ffcc22b7e0fe10769b02cd7: Status 404 returned error can't find the container with id 1bc02b6bbd9ad932cba8ba6a1f2761c6d15ad7872ffcc22b7e0fe10769b02cd7 Oct 30 00:44:23.665284 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.665250 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" event={"ID":"4e777165-f922-436a-a9a5-d21765c8ff20","Type":"ContainerStarted","Data":"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58"} Oct 30 00:44:23.665772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.665294 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" event={"ID":"4e777165-f922-436a-a9a5-d21765c8ff20","Type":"ContainerStarted","Data":"1bc02b6bbd9ad932cba8ba6a1f2761c6d15ad7872ffcc22b7e0fe10769b02cd7"} Oct 30 00:44:23.681280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.681252 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:44:23.706790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.706745 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podStartSLOduration=1.7067322969999998 podStartE2EDuration="1.706732297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:23.705112406 +0000 UTC m=+715.945347520" watchObservedRunningTime="2026-10-30 00:44:23.706732297 +0000 UTC m=+715.946967437" Oct 30 00:44:23.723372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:23.723342 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:24.392231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:24.392185 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:25.928837 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:25.928798 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:27.360602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:27.360563 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:27.688744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:27.688666 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:29.013889 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:44:29.010352 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:44:29.489816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:29.489780 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:30.527088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.527054 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn"] Oct 30 00:44:30.531496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.531478 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.550713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.550689 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn"] Oct 30 00:44:30.559648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.559627 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:44:30.559878 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.559845 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:44:30.560011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.559974 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:44:30.560201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.560177 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:44:30.626085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.626056 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.626217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.626167 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tsz\" (UniqueName: \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.626298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.626278 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.699386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.699361 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:30.726959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.726938 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.727087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.726974 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tsz\" (UniqueName: \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.727087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.727007 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.727693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.727673 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.729230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.729210 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.749671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.749644 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tsz\" (UniqueName: \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\") pod \"maas-api-revoke-sub-f4e3f943-tdtbn\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:30.843953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:30.843859 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:31.141665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:31.141620 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn"] Oct 30 00:44:31.144621 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:31.144589 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7bca6c_c5f6_4ce7_99ee_83920d1fdfae.slice/crio-887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c WatchSource:0}: Error finding container 887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c: Status 404 returned error can't find the container with id 887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c Oct 30 00:44:31.171389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:31.171365 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:31.711830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:31.711801 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" event={"ID":"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae","Type":"ContainerStarted","Data":"887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c"} Oct 30 00:44:32.723604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:32.723571 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:32.738078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:32.738055 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:32.817185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:32.817126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:33.194715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.194676 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 00:44:33.206210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.206185 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.218454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.218427 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 00:44:33.227398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.227374 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"kube-root-ca.crt\"" Oct 30 00:44:33.227398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.227370 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"openshift-service-ca.crt\"" Oct 30 00:44:33.227579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.227409 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"default-dockercfg-mtkh7\"" Oct 30 00:44:33.231991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.231969 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"premium-w6-021d6d-kserve-self-signed-certs\"" Oct 30 00:44:33.252571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252545 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.252571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252601 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xksl9\" (UniqueName: \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.252769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252639 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.252769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252709 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.252904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252826 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.252953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.252945 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.277040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.277012 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:33.354369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354332 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354433 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354491 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksl9\" (UniqueName: \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354525 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354551 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354587 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354706 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.354830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.354796 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.355038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.355014 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.356607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.356587 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.356912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.356895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.379430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.379399 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksl9\" (UniqueName: \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\") pod \"premium-w6-021d6d-kserve-69474b4b9f-qmb9s\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.523563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.523539 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:33.726057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.725984 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" event={"ID":"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae","Type":"ContainerStarted","Data":"db29c376d356ad89eb9b29262b7d239c0319ff1b4dd7e4f6b892d761347bf24a"} Oct 30 00:44:33.734612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.734588 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:44:33.765200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.765156 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" podStartSLOduration=1.484230071 podStartE2EDuration="3.765141364s" totalImagesPullingTime="2.280911293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:33.764322734 +0000 UTC m=+726.004557850" watchObservedRunningTime="2026-10-30 00:44:33.765141364 +0000 UTC m=+726.005376477" Oct 30 00:44:33.819735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:33.819699 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 00:44:33.822330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:33.822298 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c8a428a_e570_46b0_a82d_838d0c76cad0.slice/crio-f72cb4fed4332d932ae250819652240be24335f4bc96b72e966c699ad3876a37 WatchSource:0}: Error finding container f72cb4fed4332d932ae250819652240be24335f4bc96b72e966c699ad3876a37: Status 404 returned error can't find the container with id f72cb4fed4332d932ae250819652240be24335f4bc96b72e966c699ad3876a37 Oct 30 00:44:34.382254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.382228 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:34.732388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.732353 5938 generic.go:356] "Generic (PLEG): container finished" podID="0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" containerID="db29c376d356ad89eb9b29262b7d239c0319ff1b4dd7e4f6b892d761347bf24a" exitCode=0 Oct 30 00:44:34.732849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.732439 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" event={"ID":"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae","Type":"ContainerDied","Data":"db29c376d356ad89eb9b29262b7d239c0319ff1b4dd7e4f6b892d761347bf24a"} Oct 30 00:44:34.734104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.734081 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" event={"ID":"7c8a428a-e570-46b0-a82d-838d0c76cad0","Type":"ContainerStarted","Data":"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e"} Oct 30 00:44:34.734214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.734116 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" event={"ID":"7c8a428a-e570-46b0-a82d-838d0c76cad0","Type":"ContainerStarted","Data":"f72cb4fed4332d932ae250819652240be24335f4bc96b72e966c699ad3876a37"} Oct 30 00:44:34.815408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:34.815352 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podStartSLOduration=1.8153343560000001 podStartE2EDuration="1.815334356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:34.814347008 +0000 UTC m=+727.054582139" watchObservedRunningTime="2026-10-30 00:44:34.815334356 +0000 UTC m=+727.055569470" Oct 30 00:44:35.427616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.427582 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:35.524382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.524360 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:35.901232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.901206 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:35.982498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.982464 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\") pod \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " Oct 30 00:44:35.982498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.982502 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\" (UniqueName: \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\") pod \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " Oct 30 00:44:35.982707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.982613 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\") pod \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\" (UID: \"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae\") " Oct 30 00:44:35.983015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.982980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca" pod "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" (UID: "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:35.984952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.984925 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca" pod "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" (UID: "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:35.985053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:35.985015 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz" pod "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" (UID: "0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae"). InnerVolumeSpecName "kube-api-access-d6tsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:36.084206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.084162 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:36.084206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.084199 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tsz\" (UniqueName: \"kubernetes.io/projected/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-kube-api-access-d6tsz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:36.084397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.084215 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:36.158850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.158814 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:36.749366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.749323 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" event={"ID":"0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae","Type":"ContainerDied","Data":"887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c"} Oct 30 00:44:36.749366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.749356 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn" Oct 30 00:44:36.749575 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:36.749383 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="887c9ec5a931d67104cc42c5e25b9ff983a0e5b8b4641790eb6ab62bb214891c" Oct 30 00:44:37.781687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:37.781635 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:39.446925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:39.446824 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:41.000068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:41.000018 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:42.721997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:42.721966 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:43.524018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:43.523987 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:43.536448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:43.536421 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:43.797534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:43.797462 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:44:44.357529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.357494 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 00:44:44.358197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.358174 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" containerName="revoke-keys" Oct 30 00:44:44.358197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.358196 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" containerName="revoke-keys" Oct 30 00:44:44.358380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.358339 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" containerName="revoke-keys" Oct 30 00:44:44.361455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.361437 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.379677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.379652 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 00:44:44.388653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.388628 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1a11cf\"/\"premium-w2-1a11cf-kserve-self-signed-certs\"" Oct 30 00:44:44.445979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.445954 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:44.480164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7msl\" (UniqueName: \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.480260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.480260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480211 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.480419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.480458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480429 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.480458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.480447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.581940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.581905 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.581940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.581946 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.581968 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582059 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7msl\" (UniqueName: \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582110 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582134 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582346 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582502 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.582564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.582545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.584270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.584248 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.584533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.584516 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.609206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.609148 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7msl\" (UniqueName: \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\") pod \"premium-w2-1a11cf-kserve-6954dd7bcf-slhqc\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.643745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.643720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:44.676003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.675980 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:44.866709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.866640 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n"] Oct 30 00:44:44.870481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.870464 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:44.885525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.885493 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:44:44.885525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.885497 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:44:44.885735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.885585 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:44:44.885795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.885732 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:44:44.888582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.888559 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n"] Oct 30 00:44:44.971347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.971327 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 00:44:44.973985 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:44.973957 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f8f9c58_1e95_40cd_911f_7fdaa5052c58.slice/crio-6003bf2b11109c34df96a747078e20962232051367e97c6c20106aba5b35bdb6 WatchSource:0}: Error finding container 6003bf2b11109c34df96a747078e20962232051367e97c6c20106aba5b35bdb6: Status 404 returned error can't find the container with id 6003bf2b11109c34df96a747078e20962232051367e97c6c20106aba5b35bdb6 Oct 30 00:44:44.986084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.986059 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:44.986176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.986105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:44.986274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:44.986252 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7cd\" (UniqueName: \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.087072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.087043 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.087181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.087100 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.087241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.087219 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7cd\" (UniqueName: \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.087762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.087743 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.089181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.089162 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.128600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.128535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7cd\" (UniqueName: \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\") pod \"maas-api-revoke-sub-6c821316-x7j2n\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.184204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.184179 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:45.481851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.481828 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n"] Oct 30 00:44:45.485286 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:45.485257 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2fbfcc_5218_4d5a_a512_161782d706a3.slice/crio-e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704 WatchSource:0}: Error finding container e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704: Status 404 returned error can't find the container with id e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704 Oct 30 00:44:45.801882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.801844 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" event={"ID":"6f8f9c58-1e95-40cd-911f-7fdaa5052c58","Type":"ContainerStarted","Data":"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2"} Oct 30 00:44:45.802161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.802138 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" event={"ID":"6f8f9c58-1e95-40cd-911f-7fdaa5052c58","Type":"ContainerStarted","Data":"6003bf2b11109c34df96a747078e20962232051367e97c6c20106aba5b35bdb6"} Oct 30 00:44:45.803523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.803502 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" event={"ID":"2b2fbfcc-5218-4d5a-a512-161782d706a3","Type":"ContainerStarted","Data":"e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704"} Oct 30 00:44:45.845312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:45.845267 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podStartSLOduration=1.84525311 podStartE2EDuration="1.84525311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:45.844176359 +0000 UTC m=+738.084411473" watchObservedRunningTime="2026-10-30 00:44:45.84525311 +0000 UTC m=+738.085488224" Oct 30 00:44:46.315082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:46.315050 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:46.677043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:46.677015 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:46.820888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:46.820421 5938 generic.go:356] "Generic (PLEG): container finished" podID="2b2fbfcc-5218-4d5a-a512-161782d706a3" containerID="af5ad5a469341653b68484b2ec89ecfca073e5f2a5c4edab7a44ff967f73c430" exitCode=0 Oct 30 00:44:46.820888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:46.820571 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" event={"ID":"2b2fbfcc-5218-4d5a-a512-161782d706a3","Type":"ContainerDied","Data":"af5ad5a469341653b68484b2ec89ecfca073e5f2a5c4edab7a44ff967f73c430"} Oct 30 00:44:47.890913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:47.890885 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:47.983253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:47.983229 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:48.121186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.121140 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\" (UniqueName: \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\") pod \"2b2fbfcc-5218-4d5a-a512-161782d706a3\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " Oct 30 00:44:48.121329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.121289 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\") pod \"2b2fbfcc-5218-4d5a-a512-161782d706a3\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " Oct 30 00:44:48.121391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.121358 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\") pod \"2b2fbfcc-5218-4d5a-a512-161782d706a3\" (UID: \"2b2fbfcc-5218-4d5a-a512-161782d706a3\") " Oct 30 00:44:48.121639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.121616 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca" pod "2b2fbfcc-5218-4d5a-a512-161782d706a3" (UID: "2b2fbfcc-5218-4d5a-a512-161782d706a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:48.123294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.123262 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd" pod "2b2fbfcc-5218-4d5a-a512-161782d706a3" (UID: "2b2fbfcc-5218-4d5a-a512-161782d706a3"). InnerVolumeSpecName "kube-api-access-vq7cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:48.123398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.123336 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca" pod "2b2fbfcc-5218-4d5a-a512-161782d706a3" (UID: "2b2fbfcc-5218-4d5a-a512-161782d706a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:48.222545 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.222518 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7cd\" (UniqueName: \"kubernetes.io/projected/2b2fbfcc-5218-4d5a-a512-161782d706a3-kube-api-access-vq7cd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:48.222643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.222547 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:48.222643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.222564 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2fbfcc-5218-4d5a-a512-161782d706a3-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:48.832355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.832325 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" Oct 30 00:44:48.832355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.832338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n" event={"ID":"2b2fbfcc-5218-4d5a-a512-161782d706a3","Type":"ContainerDied","Data":"e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704"} Oct 30 00:44:48.832548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:48.832384 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7b3cbe17f523638cd3aecb8399a0ea2e3f6e8835e6d198ef11d44fa91b71704" Oct 30 00:44:49.548181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:49.548144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:51.076314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.076278 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z"] Oct 30 00:44:51.076787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.076772 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2b2fbfcc-5218-4d5a-a512-161782d706a3" containerName="revoke-keys" Oct 30 00:44:51.076828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.076789 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2fbfcc-5218-4d5a-a512-161782d706a3" containerName="revoke-keys" Oct 30 00:44:51.076889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.076878 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2fbfcc-5218-4d5a-a512-161782d706a3" containerName="revoke-keys" Oct 30 00:44:51.080276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.080261 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.097233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.097083 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:44:51.099123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.099093 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z"] Oct 30 00:44:51.108653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.108630 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:44:51.108795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.108711 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:44:51.108795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.108711 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:44:51.254170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.254143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7fg\" (UniqueName: \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.254326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.254190 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.254326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.254280 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.254940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.254919 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:51.355537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.355504 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7fg\" (UniqueName: \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.355700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.355551 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.355700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.355585 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.356342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.356322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.357926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.357907 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.377959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.377934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7fg\" (UniqueName: \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\") pod \"maas-api-revoke-sub-ae8f8554-9s74z\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.393388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.393362 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:51.693064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.693041 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z"] Oct 30 00:44:51.695373 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:51.695345 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c3fa73_9355_4821_b0b3_7ddfe2970beb.slice/crio-1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca WatchSource:0}: Error finding container 1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca: Status 404 returned error can't find the container with id 1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca Oct 30 00:44:51.853235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:51.853207 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" event={"ID":"96c3fa73-9355-4821-b0b3-7ddfe2970beb","Type":"ContainerStarted","Data":"1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca"} Oct 30 00:44:52.864628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:52.864258 5938 generic.go:356] "Generic (PLEG): container finished" podID="96c3fa73-9355-4821-b0b3-7ddfe2970beb" containerID="5558a968d71c74b6448ae5cd45e64c1299c06846da03a45bbe951066cc376f72" exitCode=0 Oct 30 00:44:52.864628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:52.864474 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" event={"ID":"96c3fa73-9355-4821-b0b3-7ddfe2970beb","Type":"ContainerDied","Data":"5558a968d71c74b6448ae5cd45e64c1299c06846da03a45bbe951066cc376f72"} Oct 30 00:44:52.898225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:52.898190 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:54.014122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.014094 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:54.081260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.081227 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\") pod \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " Oct 30 00:44:54.081393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.081279 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\") pod \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " Oct 30 00:44:54.081393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.081347 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\" (UniqueName: \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\") pod \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\" (UID: \"96c3fa73-9355-4821-b0b3-7ddfe2970beb\") " Oct 30 00:44:54.081665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.081642 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca" pod "96c3fa73-9355-4821-b0b3-7ddfe2970beb" (UID: "96c3fa73-9355-4821-b0b3-7ddfe2970beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:54.081814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.081798 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:54.083375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.083345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca" pod "96c3fa73-9355-4821-b0b3-7ddfe2970beb" (UID: "96c3fa73-9355-4821-b0b3-7ddfe2970beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:54.083481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.083455 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg" pod "96c3fa73-9355-4821-b0b3-7ddfe2970beb" (UID: "96c3fa73-9355-4821-b0b3-7ddfe2970beb"). InnerVolumeSpecName "kube-api-access-dx7fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:54.182668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.182605 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7fg\" (UniqueName: \"kubernetes.io/projected/96c3fa73-9355-4821-b0b3-7ddfe2970beb-kube-api-access-dx7fg\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:54.182668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.182632 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c3fa73-9355-4821-b0b3-7ddfe2970beb-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:44:54.590829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.590745 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:54.676174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.676141 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:54.688637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.688612 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:54.878032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.878007 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" Oct 30 00:44:54.878164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.878051 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z" event={"ID":"96c3fa73-9355-4821-b0b3-7ddfe2970beb","Type":"ContainerDied","Data":"1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca"} Oct 30 00:44:54.878164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.878089 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b9acdae6d44ba8b87d3166bb5f0b9b0d08972152dec719a27fc6f5f888820ca" Oct 30 00:44:54.890330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:54.890309 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:44:56.254695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:56.254662 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:57.354610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.354578 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7"] Oct 30 00:44:57.355062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.355048 5938 cpu_manager.go:414] "containerMap: removing container" podUID="96c3fa73-9355-4821-b0b3-7ddfe2970beb" containerName="revoke-keys" Oct 30 00:44:57.355120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.355063 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c3fa73-9355-4821-b0b3-7ddfe2970beb" containerName="revoke-keys" Oct 30 00:44:57.355173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.355164 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c3fa73-9355-4821-b0b3-7ddfe2970beb" containerName="revoke-keys" Oct 30 00:44:57.358419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.358403 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.378205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.378176 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7"] Oct 30 00:44:57.386014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.385992 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:44:57.386014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.386005 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:44:57.388183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.388164 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:44:57.388275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.388230 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:44:57.414064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.414041 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.414165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.414090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.414235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.414211 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4w56\" (UniqueName: \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.515043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.515019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.515182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.515073 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4w56\" (UniqueName: \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.515247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.515182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.515755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.515726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.517426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.517407 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.539884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.539839 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4w56\" (UniqueName: \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\") pod \"maas-api-revoke-sub-7bedad60-drxs7\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.671538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.671486 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:44:57.850936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.850891 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:57.968585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:57.968549 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7"] Oct 30 00:44:57.971132 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:44:57.971096 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c57169_dc49_44f5_bceb_8de46ecaf21c.slice/crio-b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774 WatchSource:0}: Error finding container b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774: Status 404 returned error can't find the container with id b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774 Oct 30 00:44:58.833912 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:44:58.833858 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:44:58.901612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:58.901584 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" event={"ID":"b8c57169-dc49-44f5-bceb-8de46ecaf21c","Type":"ContainerStarted","Data":"4e5c67b0b2e6f9cc2dbffae65cbb7614592712e95e91f61c03ee155d91c3ecc3"} Oct 30 00:44:58.901746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:58.901624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" event={"ID":"b8c57169-dc49-44f5-bceb-8de46ecaf21c","Type":"ContainerStarted","Data":"b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774"} Oct 30 00:44:58.940410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:58.940369 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" podStartSLOduration=1.940355296 podStartE2EDuration="1.940355296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:58.939308306 +0000 UTC m=+751.179543419" watchObservedRunningTime="2026-10-30 00:44:58.940355296 +0000 UTC m=+751.180590451" Oct 30 00:44:59.559338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:59.559306 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:59.937886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:59.937138 5938 generic.go:356] "Generic (PLEG): container finished" podID="b8c57169-dc49-44f5-bceb-8de46ecaf21c" containerID="4e5c67b0b2e6f9cc2dbffae65cbb7614592712e95e91f61c03ee155d91c3ecc3" exitCode=0 Oct 30 00:44:59.937886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:44:59.937391 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" event={"ID":"b8c57169-dc49-44f5-bceb-8de46ecaf21c","Type":"ContainerDied","Data":"4e5c67b0b2e6f9cc2dbffae65cbb7614592712e95e91f61c03ee155d91c3ecc3"} Oct 30 00:45:00.164886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.164313 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:00.168881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.168560 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:00.168881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.168703 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.183541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.172619 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.200591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.193523 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:00.202713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.202064 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:00.202713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.202115 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:00.202713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.202213 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.224905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.205287 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:00.234932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.232562 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.282784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.266649 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877hx\" (UniqueName: \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.291497 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.290152 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.291718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.291693 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9td\" (UniqueName: \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.291899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.291879 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qnf\" (UniqueName: \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.292063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.292043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.292209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.292192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.292360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.292339 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.292517 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.292500 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcnk\" (UniqueName: \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.292977 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:00.274456 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bb7263_2ad1_474f_a932_b63edc372453.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:45:00.304068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.303978 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:00.319842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.319818 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:00.396630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877hx\" (UniqueName: \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.396795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.396795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9td\" (UniqueName: \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.396795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396721 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qnf\" (UniqueName: \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.396795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396755 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.396795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396787 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.397127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.397127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.396899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcnk\" (UniqueName: \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.398099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.398068 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.398965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.398287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.399093 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.398908 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.399203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.399135 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.423114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.422306 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcnk\" (UniqueName: \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:00.423114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.422822 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qnf\" (UniqueName: \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\") pod \"maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.424061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.423610 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877hx\" (UniqueName: \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.424433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.424394 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9td\" (UniqueName: \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.522932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.522829 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:00.542398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.542367 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:00.639108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.630501 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:00.676281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:00.649047 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:01.290884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.286666 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:01.444031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.443627 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:01.512628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.512243 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:01.861906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.860462 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:45:01.972235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.972205 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\") pod \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " Oct 30 00:45:01.972406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.972295 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\" (UniqueName: \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\") pod \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " Oct 30 00:45:01.972406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.972402 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\") pod \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\" (UID: \"b8c57169-dc49-44f5-bceb-8de46ecaf21c\") " Oct 30 00:45:01.972767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.972729 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca" pod "b8c57169-dc49-44f5-bceb-8de46ecaf21c" (UID: "b8c57169-dc49-44f5-bceb-8de46ecaf21c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:01.973228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.973204 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:01.974420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.974392 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca" pod "b8c57169-dc49-44f5-bceb-8de46ecaf21c" (UID: "b8c57169-dc49-44f5-bceb-8de46ecaf21c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:01.975009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.974975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56" pod "b8c57169-dc49-44f5-bceb-8de46ecaf21c" (UID: "b8c57169-dc49-44f5-bceb-8de46ecaf21c"). InnerVolumeSpecName "kube-api-access-n4w56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:01.978417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.978394 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" event={"ID":"306e16db-7423-43d4-b1ad-f4ddbf949200","Type":"ContainerStarted","Data":"ca8ddf1a608373e370a76d0565e66e72891f3ee951f1b5c94082c48976a942a1"} Oct 30 00:45:01.980172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.980150 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" Oct 30 00:45:01.980280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.980154 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7" event={"ID":"b8c57169-dc49-44f5-bceb-8de46ecaf21c","Type":"ContainerDied","Data":"b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774"} Oct 30 00:45:01.980280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.980287 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9a5ee83795e6fb044b2c4b3b94f227a4ea48cf2b8b16961ac0e287a03293774" Oct 30 00:45:01.982251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:01.982216 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" event={"ID":"1ab30327-397d-4598-b28c-da5f41c49711","Type":"ContainerStarted","Data":"6de76aa53b94f58865a38745f2d5d3583fa60db70e1b64da2fa7251b28a76a76"} Oct 30 00:45:02.013155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.013134 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:02.037181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.037160 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:02.074337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.074311 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c57169-dc49-44f5-bceb-8de46ecaf21c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:02.074337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.074336 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4w56\" (UniqueName: \"kubernetes.io/projected/b8c57169-dc49-44f5-bceb-8de46ecaf21c-kube-api-access-n4w56\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:02.086206 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:02.086177 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bb7263_2ad1_474f_a932_b63edc372453.slice/crio-8ec249dba248c0673fb38db26b5f5ccbb90eab62f1de92280341a632f2efae7a WatchSource:0}: Error finding container 8ec249dba248c0673fb38db26b5f5ccbb90eab62f1de92280341a632f2efae7a: Status 404 returned error can't find the container with id 8ec249dba248c0673fb38db26b5f5ccbb90eab62f1de92280341a632f2efae7a Oct 30 00:45:02.087342 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:02.087316 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5895283d_4473_490e_aee9_fa029122306c.slice/crio-aec377948549c8f095a6e5b67b5f23f792b7251e7568afdfc5ed3d8254bac723 WatchSource:0}: Error finding container aec377948549c8f095a6e5b67b5f23f792b7251e7568afdfc5ed3d8254bac723: Status 404 returned error can't find the container with id aec377948549c8f095a6e5b67b5f23f792b7251e7568afdfc5ed3d8254bac723 Oct 30 00:45:02.991283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.991233 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerStarted","Data":"4e783f5281fa461f738f218ad7d6a3cdeda6191496aa82a841dc79dc9b24ba62"} Oct 30 00:45:02.991793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.991299 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerStarted","Data":"aec377948549c8f095a6e5b67b5f23f792b7251e7568afdfc5ed3d8254bac723"} Oct 30 00:45:02.993314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.993282 5938 generic.go:356] "Generic (PLEG): container finished" podID="1ab30327-397d-4598-b28c-da5f41c49711" containerID="e92fdcea359b4624bfb47c91be6848b3b407ac5db9ed5a5e21e18087d5a01d45" exitCode=60 Oct 30 00:45:02.993440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.993319 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" event={"ID":"1ab30327-397d-4598-b28c-da5f41c49711","Type":"ContainerDied","Data":"e92fdcea359b4624bfb47c91be6848b3b407ac5db9ed5a5e21e18087d5a01d45"} Oct 30 00:45:02.993603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.993585 5938 scope.go:122] "RemoveContainer" containerID="e92fdcea359b4624bfb47c91be6848b3b407ac5db9ed5a5e21e18087d5a01d45" Oct 30 00:45:02.995402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.995370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerStarted","Data":"b2ffdfad0b4e2f2e9954331d23fa486bfa5b394b7ab5a0b3f8cc9aa92dd4f5bf"} Oct 30 00:45:02.995511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.995417 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerStarted","Data":"8ec249dba248c0673fb38db26b5f5ccbb90eab62f1de92280341a632f2efae7a"} Oct 30 00:45:02.997376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.997356 5938 generic.go:356] "Generic (PLEG): container finished" podID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerID="fd2e277678f8f5c02d34b05227fe30e7b12c0429a13de2943fc47a84ce675463" exitCode=60 Oct 30 00:45:02.997466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.997400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" event={"ID":"306e16db-7423-43d4-b1ad-f4ddbf949200","Type":"ContainerDied","Data":"fd2e277678f8f5c02d34b05227fe30e7b12c0429a13de2943fc47a84ce675463"} Oct 30 00:45:02.997638 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:02.997621 5938 scope.go:122] "RemoveContainer" containerID="fd2e277678f8f5c02d34b05227fe30e7b12c0429a13de2943fc47a84ce675463" Oct 30 00:45:03.075816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:03.075774 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" podStartSLOduration=3.075759759 podStartE2EDuration="3.075759759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.035602258 +0000 UTC m=+755.275837374" watchObservedRunningTime="2026-10-30 00:45:03.075759759 +0000 UTC m=+755.315994871" Oct 30 00:45:03.114151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:03.114103 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" podStartSLOduration=3.114086395 podStartE2EDuration="3.114086395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.111191773 +0000 UTC m=+755.351426898" watchObservedRunningTime="2026-10-30 00:45:03.114086395 +0000 UTC m=+755.354321509" Oct 30 00:45:03.118164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:03.118141 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:04.004756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.004731 5938 generic.go:356] "Generic (PLEG): container finished" podID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerID="0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c" exitCode=60 Oct 30 00:45:04.005104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.004772 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" event={"ID":"306e16db-7423-43d4-b1ad-f4ddbf949200","Type":"ContainerDied","Data":"0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c"} Oct 30 00:45:04.005104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.004835 5938 scope.go:122] "RemoveContainer" containerID="fd2e277678f8f5c02d34b05227fe30e7b12c0429a13de2943fc47a84ce675463" Oct 30 00:45:04.005104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.005034 5938 scope.go:122] "RemoveContainer" containerID="0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c" Oct 30 00:45:04.005283 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:04.005247 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5_odh-ai-gateway-infra(306e16db-7423-43d4-b1ad-f4ddbf949200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" Oct 30 00:45:04.006500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.006480 5938 generic.go:356] "Generic (PLEG): container finished" podID="5895283d-4473-490e-aee9-fa029122306c" containerID="4e783f5281fa461f738f218ad7d6a3cdeda6191496aa82a841dc79dc9b24ba62" exitCode=60 Oct 30 00:45:04.006572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.006530 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerDied","Data":"4e783f5281fa461f738f218ad7d6a3cdeda6191496aa82a841dc79dc9b24ba62"} Oct 30 00:45:04.006712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.006695 5938 scope.go:122] "RemoveContainer" containerID="4e783f5281fa461f738f218ad7d6a3cdeda6191496aa82a841dc79dc9b24ba62" Oct 30 00:45:04.008628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.008597 5938 generic.go:356] "Generic (PLEG): container finished" podID="1ab30327-397d-4598-b28c-da5f41c49711" containerID="02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21" exitCode=60 Oct 30 00:45:04.008720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.008675 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" event={"ID":"1ab30327-397d-4598-b28c-da5f41c49711","Type":"ContainerDied","Data":"02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21"} Oct 30 00:45:04.008893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.008853 5938 scope.go:122] "RemoveContainer" containerID="02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21" Oct 30 00:45:04.009445 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:04.009423 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs_odh-ai-gateway-infra(1ab30327-397d-4598-b28c-da5f41c49711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" podUID="1ab30327-397d-4598-b28c-da5f41c49711" Oct 30 00:45:04.010304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.010284 5938 generic.go:356] "Generic (PLEG): container finished" podID="72bb7263-2ad1-474f-a932-b63edc372453" containerID="b2ffdfad0b4e2f2e9954331d23fa486bfa5b394b7ab5a0b3f8cc9aa92dd4f5bf" exitCode=60 Oct 30 00:45:04.010382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.010342 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerDied","Data":"b2ffdfad0b4e2f2e9954331d23fa486bfa5b394b7ab5a0b3f8cc9aa92dd4f5bf"} Oct 30 00:45:04.010492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.010479 5938 scope.go:122] "RemoveContainer" containerID="b2ffdfad0b4e2f2e9954331d23fa486bfa5b394b7ab5a0b3f8cc9aa92dd4f5bf" Oct 30 00:45:04.048927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.048904 5938 scope.go:122] "RemoveContainer" containerID="e92fdcea359b4624bfb47c91be6848b3b407ac5db9ed5a5e21e18087d5a01d45" Oct 30 00:45:04.822724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:04.822684 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:05.018237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.018216 5938 scope.go:122] "RemoveContainer" containerID="0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c" Oct 30 00:45:05.018626 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:05.018441 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5_odh-ai-gateway-infra(306e16db-7423-43d4-b1ad-f4ddbf949200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" Oct 30 00:45:05.019930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.019907 5938 generic.go:356] "Generic (PLEG): container finished" podID="5895283d-4473-490e-aee9-fa029122306c" containerID="9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f" exitCode=60 Oct 30 00:45:05.020028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.019986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerDied","Data":"9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f"} Oct 30 00:45:05.020085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.020050 5938 scope.go:122] "RemoveContainer" containerID="4e783f5281fa461f738f218ad7d6a3cdeda6191496aa82a841dc79dc9b24ba62" Oct 30 00:45:05.020207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.020193 5938 scope.go:122] "RemoveContainer" containerID="9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f" Oct 30 00:45:05.020428 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:05.020408 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt_odh-ai-gateway-infra(5895283d-4473-490e-aee9-fa029122306c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" podUID="5895283d-4473-490e-aee9-fa029122306c" Oct 30 00:45:05.022451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.022427 5938 scope.go:122] "RemoveContainer" containerID="02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21" Oct 30 00:45:05.022907 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:05.022658 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs_odh-ai-gateway-infra(1ab30327-397d-4598-b28c-da5f41c49711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" podUID="1ab30327-397d-4598-b28c-da5f41c49711" Oct 30 00:45:05.024557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.024532 5938 generic.go:356] "Generic (PLEG): container finished" podID="72bb7263-2ad1-474f-a932-b63edc372453" containerID="92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4" exitCode=60 Oct 30 00:45:05.024666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.024650 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerDied","Data":"92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4"} Oct 30 00:45:05.025034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.025016 5938 scope.go:122] "RemoveContainer" containerID="92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4" Oct 30 00:45:05.025268 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:05.025244 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b_odh-ai-gateway-infra(72bb7263-2ad1-474f-a932-b63edc372453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" podUID="72bb7263-2ad1-474f-a932-b63edc372453" Oct 30 00:45:05.031204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:05.031184 5938 scope.go:122] "RemoveContainer" containerID="b2ffdfad0b4e2f2e9954331d23fa486bfa5b394b7ab5a0b3f8cc9aa92dd4f5bf" Oct 30 00:45:06.032136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.032108 5938 scope.go:122] "RemoveContainer" containerID="9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f" Oct 30 00:45:06.032549 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:06.032345 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt_odh-ai-gateway-infra(5895283d-4473-490e-aee9-fa029122306c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" podUID="5895283d-4473-490e-aee9-fa029122306c" Oct 30 00:45:06.033768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.033748 5938 scope.go:122] "RemoveContainer" containerID="92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4" Oct 30 00:45:06.033973 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:06.033952 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b_odh-ai-gateway-infra(72bb7263-2ad1-474f-a932-b63edc372453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" podUID="72bb7263-2ad1-474f-a932-b63edc372453" Oct 30 00:45:06.493390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.493338 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:06.941953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.941926 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz"] Oct 30 00:45:06.942389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.942377 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b8c57169-dc49-44f5-bceb-8de46ecaf21c" containerName="revoke-keys" Oct 30 00:45:06.942433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.942390 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c57169-dc49-44f5-bceb-8de46ecaf21c" containerName="revoke-keys" Oct 30 00:45:06.942505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.942496 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c57169-dc49-44f5-bceb-8de46ecaf21c" containerName="revoke-keys" Oct 30 00:45:06.945341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.945323 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:06.961035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.961015 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:45:06.975774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:06.975749 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz"] Oct 30 00:45:07.028191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.028163 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kzq\" (UniqueName: \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.028318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.028294 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.028361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.028353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.129804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.129776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.130233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.129832 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.130233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.129956 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kzq\" (UniqueName: \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.130604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.130575 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.132228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.132210 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.155680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.155651 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kzq\" (UniqueName: \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\") pod \"maas-api-revoke-sub-d4e9dc7d-zxqqz\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.257442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.257385 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:07.586102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:07.584954 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz"] Oct 30 00:45:07.593413 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:07.593376 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36353c1c_1c47_4bd9_804b_fe862dcbb322.slice/crio-d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb WatchSource:0}: Error finding container d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb: Status 404 returned error can't find the container with id d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb Oct 30 00:45:08.046142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.046101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" event={"ID":"36353c1c-1c47-4bd9-804b-fe862dcbb322","Type":"ContainerStarted","Data":"8881b687cd943f8dac23c84399e1af5b3bb0c2e164f839936d8b970263eff5c2"} Oct 30 00:45:08.046316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.046158 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" event={"ID":"36353c1c-1c47-4bd9-804b-fe862dcbb322","Type":"ContainerStarted","Data":"d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb"} Oct 30 00:45:08.078986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.078958 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:08.084720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.084492 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" podStartSLOduration=2.084474195 podStartE2EDuration="2.084474195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:08.084142288 +0000 UTC m=+760.324377401" watchObservedRunningTime="2026-10-30 00:45:08.084474195 +0000 UTC m=+760.324709326" Oct 30 00:45:08.484014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.483977 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29"] Oct 30 00:45:08.487883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.487834 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.508201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.506508 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29"] Oct 30 00:45:08.559109 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.559080 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.559255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.559154 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4gf9\" (UniqueName: \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.559326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.559247 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.660336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.660310 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.660510 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.660492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.660550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.660536 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4gf9\" (UniqueName: \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.661148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.661126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.662845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.662819 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.684882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.684594 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4gf9\" (UniqueName: \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\") pod \"maas-api-revoke-sub-4c498d9c-76s29\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:08.883053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:08.883015 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:09.053080 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:09.053050 5938 generic.go:356] "Generic (PLEG): container finished" podID="36353c1c-1c47-4bd9-804b-fe862dcbb322" containerID="8881b687cd943f8dac23c84399e1af5b3bb0c2e164f839936d8b970263eff5c2" exitCode=0 Oct 30 00:45:09.053261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:09.053136 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" event={"ID":"36353c1c-1c47-4bd9-804b-fe862dcbb322","Type":"ContainerDied","Data":"8881b687cd943f8dac23c84399e1af5b3bb0c2e164f839936d8b970263eff5c2"} Oct 30 00:45:09.191518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:09.191493 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29"] Oct 30 00:45:09.193466 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:09.193443 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322c8f9a_8f06_40c9_84c4_9f13675d5aa1.slice/crio-2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055 WatchSource:0}: Error finding container 2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055: Status 404 returned error can't find the container with id 2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055 Oct 30 00:45:09.692594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:09.692555 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:10.060011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.059982 5938 generic.go:356] "Generic (PLEG): container finished" podID="322c8f9a-8f06-40c9-84c4-9f13675d5aa1" containerID="1998abb9f44bf879de0c33fcdb687775540f9aa58dd8846213ec45750dc4adf3" exitCode=0 Oct 30 00:45:10.060161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.060040 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" event={"ID":"322c8f9a-8f06-40c9-84c4-9f13675d5aa1","Type":"ContainerDied","Data":"1998abb9f44bf879de0c33fcdb687775540f9aa58dd8846213ec45750dc4adf3"} Oct 30 00:45:10.060161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.060087 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" event={"ID":"322c8f9a-8f06-40c9-84c4-9f13675d5aa1","Type":"ContainerStarted","Data":"2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055"} Oct 30 00:45:10.205771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.205749 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:10.279142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.279057 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\" (UniqueName: \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\") pod \"36353c1c-1c47-4bd9-804b-fe862dcbb322\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " Oct 30 00:45:10.279283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.279144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\") pod \"36353c1c-1c47-4bd9-804b-fe862dcbb322\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " Oct 30 00:45:10.279283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.279207 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\") pod \"36353c1c-1c47-4bd9-804b-fe862dcbb322\" (UID: \"36353c1c-1c47-4bd9-804b-fe862dcbb322\") " Oct 30 00:45:10.279506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.279479 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca" pod "36353c1c-1c47-4bd9-804b-fe862dcbb322" (UID: "36353c1c-1c47-4bd9-804b-fe862dcbb322"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:10.279753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.279736 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:10.281758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.281733 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca" pod "36353c1c-1c47-4bd9-804b-fe862dcbb322" (UID: "36353c1c-1c47-4bd9-804b-fe862dcbb322"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:10.281881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.281762 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq" pod "36353c1c-1c47-4bd9-804b-fe862dcbb322" (UID: "36353c1c-1c47-4bd9-804b-fe862dcbb322"). InnerVolumeSpecName "kube-api-access-h6kzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:10.380900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.380850 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6kzq\" (UniqueName: \"kubernetes.io/projected/36353c1c-1c47-4bd9-804b-fe862dcbb322-kube-api-access-h6kzq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:10.381014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:10.380902 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36353c1c-1c47-4bd9-804b-fe862dcbb322-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:11.066817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.066789 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" Oct 30 00:45:11.067307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.066833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz" event={"ID":"36353c1c-1c47-4bd9-804b-fe862dcbb322","Type":"ContainerDied","Data":"d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb"} Oct 30 00:45:11.067307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.066891 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d733d2b0968c36a9b9d19f58f66e1aabdaa9ec9832fe1ed67a016909bd7e86cb" Oct 30 00:45:11.200031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.200005 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:11.222496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.222439 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:11.289089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.289055 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\" (UniqueName: \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\") pod \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " Oct 30 00:45:11.289271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.289118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\") pod \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " Oct 30 00:45:11.289271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.289227 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\") pod \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\" (UID: \"322c8f9a-8f06-40c9-84c4-9f13675d5aa1\") " Oct 30 00:45:11.289669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.289635 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca" pod "322c8f9a-8f06-40c9-84c4-9f13675d5aa1" (UID: "322c8f9a-8f06-40c9-84c4-9f13675d5aa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:11.291228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.291198 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9" pod "322c8f9a-8f06-40c9-84c4-9f13675d5aa1" (UID: "322c8f9a-8f06-40c9-84c4-9f13675d5aa1"). InnerVolumeSpecName "kube-api-access-q4gf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:11.291325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.291245 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca" pod "322c8f9a-8f06-40c9-84c4-9f13675d5aa1" (UID: "322c8f9a-8f06-40c9-84c4-9f13675d5aa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:11.390529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.390503 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4gf9\" (UniqueName: \"kubernetes.io/projected/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-kube-api-access-q4gf9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:11.390529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.390528 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:11.390678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:11.390538 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322c8f9a-8f06-40c9-84c4-9f13675d5aa1-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:12.073715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:12.073687 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" Oct 30 00:45:12.073715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:12.073701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29" event={"ID":"322c8f9a-8f06-40c9-84c4-9f13675d5aa1","Type":"ContainerDied","Data":"2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055"} Oct 30 00:45:12.074117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:12.073751 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d84919e736526873d24e07d9e355406b4c4c996c2d27aae8b701769aedce055" Oct 30 00:45:12.735244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:12.735205 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:14.266888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:14.266593 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:14.871503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:14.869351 5938 scope.go:122] "RemoveContainer" containerID="0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c" Oct 30 00:45:15.857208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:15.857172 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:15.869713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:15.869680 5938 scope.go:122] "RemoveContainer" containerID="02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21" Oct 30 00:45:16.145394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.145366 5938 generic.go:356] "Generic (PLEG): container finished" podID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerID="4ec53da32dbd6b8e74804a43c87769b0260467930917cb31491b5f0425bb2cf8" exitCode=60 Oct 30 00:45:16.145525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.145437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" event={"ID":"306e16db-7423-43d4-b1ad-f4ddbf949200","Type":"ContainerDied","Data":"4ec53da32dbd6b8e74804a43c87769b0260467930917cb31491b5f0425bb2cf8"} Oct 30 00:45:16.145525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.145522 5938 scope.go:122] "RemoveContainer" containerID="0db9f438d31337be4a87a3c6b4b79bfc6608f02e832154126b3371a9eb04f48c" Oct 30 00:45:16.145918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.145903 5938 scope.go:122] "RemoveContainer" containerID="4ec53da32dbd6b8e74804a43c87769b0260467930917cb31491b5f0425bb2cf8" Oct 30 00:45:16.146212 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:16.146184 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5_odh-ai-gateway-infra(306e16db-7423-43d4-b1ad-f4ddbf949200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" Oct 30 00:45:16.392210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392133 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx"] Oct 30 00:45:16.392709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392677 5938 cpu_manager.go:414] "containerMap: removing container" podUID="322c8f9a-8f06-40c9-84c4-9f13675d5aa1" containerName="revoke-keys" Oct 30 00:45:16.392709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392695 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322c8f9a-8f06-40c9-84c4-9f13675d5aa1" containerName="revoke-keys" Oct 30 00:45:16.392968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392750 5938 cpu_manager.go:414] "containerMap: removing container" podUID="36353c1c-1c47-4bd9-804b-fe862dcbb322" containerName="revoke-keys" Oct 30 00:45:16.392968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392756 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36353c1c-1c47-4bd9-804b-fe862dcbb322" containerName="revoke-keys" Oct 30 00:45:16.392968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392885 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="36353c1c-1c47-4bd9-804b-fe862dcbb322" containerName="revoke-keys" Oct 30 00:45:16.392968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.392899 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="322c8f9a-8f06-40c9-84c4-9f13675d5aa1" containerName="revoke-keys" Oct 30 00:45:16.395233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.395212 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.411391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.411365 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:45:16.414913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.414857 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx"] Oct 30 00:45:16.589613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.589578 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.589782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.589682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgns\" (UniqueName: \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.589782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.589714 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.690333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.690259 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.690468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.690341 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgns\" (UniqueName: \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.690468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.690378 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.690950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.690931 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.692673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.692654 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.712624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.712596 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgns\" (UniqueName: \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\") pod \"maas-api-revoke-sub-99a845fb-h66gx\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:16.869529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:16.869499 5938 scope.go:122] "RemoveContainer" containerID="9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f" Oct 30 00:45:17.007970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.007938 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:17.153689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.153652 5938 generic.go:356] "Generic (PLEG): container finished" podID="1ab30327-397d-4598-b28c-da5f41c49711" containerID="22793c77d7132041e3afeb0a6434861e986134d18b66f90896f26770d6121136" exitCode=60 Oct 30 00:45:17.153836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.153725 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" event={"ID":"1ab30327-397d-4598-b28c-da5f41c49711","Type":"ContainerDied","Data":"22793c77d7132041e3afeb0a6434861e986134d18b66f90896f26770d6121136"} Oct 30 00:45:17.153836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.153788 5938 scope.go:122] "RemoveContainer" containerID="02851c46fe135295a258a2db331975256a1565af3fbbe88658608b222254cb21" Oct 30 00:45:17.154206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.154178 5938 scope.go:122] "RemoveContainer" containerID="22793c77d7132041e3afeb0a6434861e986134d18b66f90896f26770d6121136" Oct 30 00:45:17.154497 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:17.154469 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs_odh-ai-gateway-infra(1ab30327-397d-4598-b28c-da5f41c49711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" podUID="1ab30327-397d-4598-b28c-da5f41c49711" Oct 30 00:45:17.198948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.198921 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:17.350496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.350458 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx"] Oct 30 00:45:17.352524 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:17.352502 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ed0d70_32a6_4019_bbf1_898a55429bc5.slice/crio-0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975 WatchSource:0}: Error finding container 0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975: Status 404 returned error can't find the container with id 0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975 Oct 30 00:45:17.352995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.352976 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt"] Oct 30 00:45:17.358124 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.358105 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.367433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.367414 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:17.384473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.384450 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt"] Oct 30 00:45:17.400682 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.400645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\") pod \"306e16db-7423-43d4-b1ad-f4ddbf949200\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " Oct 30 00:45:17.400795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.400743 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\" (UniqueName: \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\") pod \"306e16db-7423-43d4-b1ad-f4ddbf949200\" (UID: \"306e16db-7423-43d4-b1ad-f4ddbf949200\") " Oct 30 00:45:17.401154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.401073 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.401260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.401218 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zhh\" (UniqueName: \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.401344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.401324 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.401676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.401650 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca" pod "306e16db-7423-43d4-b1ad-f4ddbf949200" (UID: "306e16db-7423-43d4-b1ad-f4ddbf949200"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:17.403533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.403513 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td" pod "306e16db-7423-43d4-b1ad-f4ddbf949200" (UID: "306e16db-7423-43d4-b1ad-f4ddbf949200"). InnerVolumeSpecName "kube-api-access-cb9td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:17.502214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.502399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502251 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zhh\" (UniqueName: \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.502399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.502399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502373 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306e16db-7423-43d4-b1ad-f4ddbf949200-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:17.502399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502383 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9td\" (UniqueName: \"kubernetes.io/projected/306e16db-7423-43d4-b1ad-f4ddbf949200-kube-api-access-cb9td\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:17.502958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.502932 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.585111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.585018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.585265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.585232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zhh\" (UniqueName: \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\") pod \"maas-api-revoke-sub-04d470fe-l48bt\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.600808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.599595 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:17.687519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.687487 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:17.992884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:17.992848 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt"] Oct 30 00:45:17.994588 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:17.994566 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1199f95a_06a7_41c1_bf0e_876eaf9790b8.slice/crio-5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7 WatchSource:0}: Error finding container 5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7: Status 404 returned error can't find the container with id 5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7 Oct 30 00:45:18.162975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.162896 5938 generic.go:356] "Generic (PLEG): container finished" podID="5895283d-4473-490e-aee9-fa029122306c" containerID="00b14a492b5939bb3165f7c0d59fbb14097b9a1fde7488f006f6a8210217d21d" exitCode=60 Oct 30 00:45:18.162975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.162957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerDied","Data":"00b14a492b5939bb3165f7c0d59fbb14097b9a1fde7488f006f6a8210217d21d"} Oct 30 00:45:18.163217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.163018 5938 scope.go:122] "RemoveContainer" containerID="9620511b5a04784ec9f94f323064e43550287c48c46fdf14c7b52694a4f5574f" Oct 30 00:45:18.163430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.163410 5938 scope.go:122] "RemoveContainer" containerID="00b14a492b5939bb3165f7c0d59fbb14097b9a1fde7488f006f6a8210217d21d" Oct 30 00:45:18.163755 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:18.163726 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt_odh-ai-gateway-infra(5895283d-4473-490e-aee9-fa029122306c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" podUID="5895283d-4473-490e-aee9-fa029122306c" Oct 30 00:45:18.165877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.165837 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" event={"ID":"1199f95a-06a7-41c1-bf0e-876eaf9790b8","Type":"ContainerStarted","Data":"5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7"} Oct 30 00:45:18.167090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.167071 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" event={"ID":"306e16db-7423-43d4-b1ad-f4ddbf949200","Type":"ContainerDied","Data":"ca8ddf1a608373e370a76d0565e66e72891f3ee951f1b5c94082c48976a942a1"} Oct 30 00:45:18.167188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.167078 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5" Oct 30 00:45:18.168834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.168758 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" event={"ID":"f8ed0d70-32a6-4019-bbf1-898a55429bc5","Type":"ContainerStarted","Data":"afd175218fe02c946b1dd01809859c18c632e67067e87ea9fa43ce694d56178f"} Oct 30 00:45:18.168834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.168802 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" event={"ID":"f8ed0d70-32a6-4019-bbf1-898a55429bc5","Type":"ContainerStarted","Data":"0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975"} Oct 30 00:45:18.175216 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.175180 5938 scope.go:122] "RemoveContainer" containerID="4ec53da32dbd6b8e74804a43c87769b0260467930917cb31491b5f0425bb2cf8" Oct 30 00:45:18.206763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.206735 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:18.288928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.288900 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:18.298628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.298526 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888685-8hrq5"] Oct 30 00:45:18.331196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.331144 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" podStartSLOduration=2.331128595 podStartE2EDuration="2.331128595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:18.33057266 +0000 UTC m=+770.570807773" watchObservedRunningTime="2026-10-30 00:45:18.331128595 +0000 UTC m=+770.571363709" Oct 30 00:45:18.391113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.391089 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:18.415433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.415376 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\" (UniqueName: \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\") pod \"1ab30327-397d-4598-b28c-da5f41c49711\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " Oct 30 00:45:18.415433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.415428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\") pod \"1ab30327-397d-4598-b28c-da5f41c49711\" (UID: \"1ab30327-397d-4598-b28c-da5f41c49711\") " Oct 30 00:45:18.415789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.415768 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca" pod "1ab30327-397d-4598-b28c-da5f41c49711" (UID: "1ab30327-397d-4598-b28c-da5f41c49711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:18.417229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.417208 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf" pod "1ab30327-397d-4598-b28c-da5f41c49711" (UID: "1ab30327-397d-4598-b28c-da5f41c49711"). InnerVolumeSpecName "kube-api-access-c2qnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:18.516499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.516464 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qnf\" (UniqueName: \"kubernetes.io/projected/1ab30327-397d-4598-b28c-da5f41c49711-kube-api-access-c2qnf\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:18.516499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.516498 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab30327-397d-4598-b28c-da5f41c49711-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:18.597648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.597618 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p"] Oct 30 00:45:18.598290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598267 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598284 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598293 5938 cpu_manager.go:414] "containerMap: removing container" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598299 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598315 5938 cpu_manager.go:414] "containerMap: removing container" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598321 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598344 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598349 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598357 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598362 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598369 5938 cpu_manager.go:414] "containerMap: removing container" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598375 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598465 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.598733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598474 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598484 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598491 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" containerName="cleanup" Oct 30 00:45:18.598733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.598497 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:18.600693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.600677 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.617762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.617740 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4n8\" (UniqueName: \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.617851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.617770 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.617949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.617921 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.620710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.620684 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p"] Oct 30 00:45:18.719608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.719538 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4n8\" (UniqueName: \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.719608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.719577 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.719770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.719645 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.720291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.720269 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.721813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.721792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.747038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.747017 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4n8\" (UniqueName: \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\") pod \"maas-api-revoke-sub-bd8cda2a-cbl9p\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:18.868791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.868764 5938 scope.go:122] "RemoveContainer" containerID="92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4" Oct 30 00:45:18.912464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:18.912433 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:19.177410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.177387 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" Oct 30 00:45:19.177853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.177405 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs" event={"ID":"1ab30327-397d-4598-b28c-da5f41c49711","Type":"ContainerDied","Data":"6de76aa53b94f58865a38745f2d5d3583fa60db70e1b64da2fa7251b28a76a76"} Oct 30 00:45:19.177853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.177463 5938 scope.go:122] "RemoveContainer" containerID="22793c77d7132041e3afeb0a6434861e986134d18b66f90896f26770d6121136" Oct 30 00:45:19.179272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.179238 5938 generic.go:356] "Generic (PLEG): container finished" podID="1199f95a-06a7-41c1-bf0e-876eaf9790b8" containerID="55ce76580a892001d53d284101b83f072fc5e727e7b609a7b1410e940968d81c" exitCode=0 Oct 30 00:45:19.179367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.179297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" event={"ID":"1199f95a-06a7-41c1-bf0e-876eaf9790b8","Type":"ContainerDied","Data":"55ce76580a892001d53d284101b83f072fc5e727e7b609a7b1410e940968d81c"} Oct 30 00:45:19.182047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.182021 5938 generic.go:356] "Generic (PLEG): container finished" podID="f8ed0d70-32a6-4019-bbf1-898a55429bc5" containerID="afd175218fe02c946b1dd01809859c18c632e67067e87ea9fa43ce694d56178f" exitCode=0 Oct 30 00:45:19.182145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.182082 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" event={"ID":"f8ed0d70-32a6-4019-bbf1-898a55429bc5","Type":"ContainerDied","Data":"afd175218fe02c946b1dd01809859c18c632e67067e87ea9fa43ce694d56178f"} Oct 30 00:45:19.222057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.222017 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:19.222217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.222196 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:19.230212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.230186 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:19.236769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.236681 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p"] Oct 30 00:45:19.242234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.242209 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-29be73-29888685-5vpzs"] Oct 30 00:45:19.244064 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:19.244034 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f37b6a2_bdf4_4e07_b27c_e694d9576680.slice/crio-1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334 WatchSource:0}: Error finding container 1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334: Status 404 returned error can't find the container with id 1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334 Oct 30 00:45:19.453912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.447176 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:19.539612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.538421 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\") pod \"5895283d-4473-490e-aee9-fa029122306c\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " Oct 30 00:45:19.539612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.538475 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\" (UniqueName: \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\") pod \"5895283d-4473-490e-aee9-fa029122306c\" (UID: \"5895283d-4473-490e-aee9-fa029122306c\") " Oct 30 00:45:19.539612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.539450 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca" pod "5895283d-4473-490e-aee9-fa029122306c" (UID: "5895283d-4473-490e-aee9-fa029122306c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:19.544084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.544056 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk" pod "5895283d-4473-490e-aee9-fa029122306c" (UID: "5895283d-4473-490e-aee9-fa029122306c"). InnerVolumeSpecName "kube-api-access-gzcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:19.640289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.640264 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5895283d-4473-490e-aee9-fa029122306c-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:19.640289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.640287 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzcnk\" (UniqueName: \"kubernetes.io/projected/5895283d-4473-490e-aee9-fa029122306c-kube-api-access-gzcnk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:19.875777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.875751 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab30327-397d-4598-b28c-da5f41c49711" path="/var/lib/kubelet/pods/1ab30327-397d-4598-b28c-da5f41c49711/volumes" Oct 30 00:45:19.876165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:19.876151 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306e16db-7423-43d4-b1ad-f4ddbf949200" path="/var/lib/kubelet/pods/306e16db-7423-43d4-b1ad-f4ddbf949200/volumes" Oct 30 00:45:20.190347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.190271 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" event={"ID":"5895283d-4473-490e-aee9-fa029122306c","Type":"ContainerDied","Data":"aec377948549c8f095a6e5b67b5f23f792b7251e7568afdfc5ed3d8254bac723"} Oct 30 00:45:20.190347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.190288 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt" Oct 30 00:45:20.190347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.190333 5938 scope.go:122] "RemoveContainer" containerID="00b14a492b5939bb3165f7c0d59fbb14097b9a1fde7488f006f6a8210217d21d" Oct 30 00:45:20.194490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.194467 5938 generic.go:356] "Generic (PLEG): container finished" podID="72bb7263-2ad1-474f-a932-b63edc372453" containerID="49856d737661f718fe15d2702a16bd30717875ea978d73476551bfbf505068f2" exitCode=60 Oct 30 00:45:20.194737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.194560 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerDied","Data":"49856d737661f718fe15d2702a16bd30717875ea978d73476551bfbf505068f2"} Oct 30 00:45:20.194977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.194960 5938 scope.go:122] "RemoveContainer" containerID="49856d737661f718fe15d2702a16bd30717875ea978d73476551bfbf505068f2" Oct 30 00:45:20.195206 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:20.195187 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b_odh-ai-gateway-infra(72bb7263-2ad1-474f-a932-b63edc372453)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" podUID="72bb7263-2ad1-474f-a932-b63edc372453" Oct 30 00:45:20.196548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.196525 5938 generic.go:356] "Generic (PLEG): container finished" podID="0f37b6a2-bdf4-4e07-b27c-e694d9576680" containerID="44046fb0887e202c4280c6e9c8cc1e0cf5e10338de948f2dfdbe97888e3868c8" exitCode=0 Oct 30 00:45:20.196648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.196586 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" event={"ID":"0f37b6a2-bdf4-4e07-b27c-e694d9576680","Type":"ContainerDied","Data":"44046fb0887e202c4280c6e9c8cc1e0cf5e10338de948f2dfdbe97888e3868c8"} Oct 30 00:45:20.196648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.196620 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" event={"ID":"0f37b6a2-bdf4-4e07-b27c-e694d9576680","Type":"ContainerStarted","Data":"1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334"} Oct 30 00:45:20.201601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.201586 5938 scope.go:122] "RemoveContainer" containerID="92bffb5f7781fd0b6a8a5f37cc3c9f485f41c44ac926cae1093f91ec79235fa4" Oct 30 00:45:20.345966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.345931 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:20.346709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.346688 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888685-8vgnt"] Oct 30 00:45:20.382850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.382825 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:20.405528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.405501 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:20.449098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449027 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\" (UniqueName: \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\") pod \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " Oct 30 00:45:20.449098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449079 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\" (UniqueName: \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\") pod \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " Oct 30 00:45:20.449309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449103 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\") pod \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " Oct 30 00:45:20.449309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\") pod \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " Oct 30 00:45:20.449539 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449517 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\") pod \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\" (UID: \"f8ed0d70-32a6-4019-bbf1-898a55429bc5\") " Oct 30 00:45:20.449612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.449581 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\") pod \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\" (UID: \"1199f95a-06a7-41c1-bf0e-876eaf9790b8\") " Oct 30 00:45:20.450473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.450445 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca" pod "1199f95a-06a7-41c1-bf0e-876eaf9790b8" (UID: "1199f95a-06a7-41c1-bf0e-876eaf9790b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:20.450785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.450760 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca" pod "f8ed0d70-32a6-4019-bbf1-898a55429bc5" (UID: "f8ed0d70-32a6-4019-bbf1-898a55429bc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:20.451542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.451517 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns" pod "f8ed0d70-32a6-4019-bbf1-898a55429bc5" (UID: "f8ed0d70-32a6-4019-bbf1-898a55429bc5"). InnerVolumeSpecName "kube-api-access-mpgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:20.451723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.451695 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh" pod "1199f95a-06a7-41c1-bf0e-876eaf9790b8" (UID: "1199f95a-06a7-41c1-bf0e-876eaf9790b8"). InnerVolumeSpecName "kube-api-access-78zhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:20.451723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.451706 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca" pod "f8ed0d70-32a6-4019-bbf1-898a55429bc5" (UID: "f8ed0d70-32a6-4019-bbf1-898a55429bc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:20.451922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.451898 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca" pod "1199f95a-06a7-41c1-bf0e-876eaf9790b8" (UID: "1199f95a-06a7-41c1-bf0e-876eaf9790b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:20.551053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551016 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.551053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551045 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.551053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551054 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zhh\" (UniqueName: \"kubernetes.io/projected/1199f95a-06a7-41c1-bf0e-876eaf9790b8-kube-api-access-78zhh\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.551253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551065 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgns\" (UniqueName: \"kubernetes.io/projected/f8ed0d70-32a6-4019-bbf1-898a55429bc5-kube-api-access-mpgns\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.551253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551074 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1199f95a-06a7-41c1-bf0e-876eaf9790b8-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.551253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.551082 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ed0d70-32a6-4019-bbf1-898a55429bc5-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:20.893249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893222 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt"] Oct 30 00:45:20.893709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893696 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893709 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893726 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f8ed0d70-32a6-4019-bbf1-898a55429bc5" containerName="revoke-keys" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893732 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ed0d70-32a6-4019-bbf1-898a55429bc5" containerName="revoke-keys" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893744 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893748 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893757 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1199f95a-06a7-41c1-bf0e-876eaf9790b8" containerName="revoke-keys" Oct 30 00:45:20.893763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893761 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1199f95a-06a7-41c1-bf0e-876eaf9790b8" containerName="revoke-keys" Oct 30 00:45:20.894009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893840 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1199f95a-06a7-41c1-bf0e-876eaf9790b8" containerName="revoke-keys" Oct 30 00:45:20.894009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893848 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.894009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893853 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab30327-397d-4598-b28c-da5f41c49711" containerName="cleanup" Oct 30 00:45:20.894009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893881 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ed0d70-32a6-4019-bbf1-898a55429bc5" containerName="revoke-keys" Oct 30 00:45:20.894009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.893888 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:20.897470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.897452 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:20.920735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.920712 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt"] Oct 30 00:45:20.955268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.955237 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zstr\" (UniqueName: \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:20.955372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.955297 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:20.955621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:20.955583 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.056446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.056412 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zstr\" (UniqueName: \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.056556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.056461 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.056556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.056510 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.057239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.057209 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.058755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.058732 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.090175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.090155 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zstr\" (UniqueName: \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\") pod \"maas-api-revoke-sub-d962fca6-cx8zt\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.101878 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.101829 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:21.203735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.203686 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" Oct 30 00:45:21.204067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.203685 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt" event={"ID":"1199f95a-06a7-41c1-bf0e-876eaf9790b8","Type":"ContainerDied","Data":"5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7"} Oct 30 00:45:21.204067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.203832 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b39162f9090f4f8ee5ec1d647cbb02a2e84305fe3e0e15640fdcca44700e0f7" Oct 30 00:45:21.205208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.205188 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" Oct 30 00:45:21.205208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.205194 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx" event={"ID":"f8ed0d70-32a6-4019-bbf1-898a55429bc5","Type":"ContainerDied","Data":"0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975"} Oct 30 00:45:21.205370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.205236 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c04ce63dd9f6e3c50067e881eb4b9e9ca68f6eb4af3011d3c0d6403576ac975" Oct 30 00:45:21.210387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.210368 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:21.297494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.296928 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:21.415510 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.415477 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:21.462613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.462537 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\" (UniqueName: \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\") pod \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " Oct 30 00:45:21.462764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.462668 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\") pod \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " Oct 30 00:45:21.462817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.462759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\") pod \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\" (UID: \"0f37b6a2-bdf4-4e07-b27c-e694d9576680\") " Oct 30 00:45:21.464352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.464320 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:21.464934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.464906 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca" pod "0f37b6a2-bdf4-4e07-b27c-e694d9576680" (UID: "0f37b6a2-bdf4-4e07-b27c-e694d9576680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:21.466083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.466053 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca" pod "0f37b6a2-bdf4-4e07-b27c-e694d9576680" (UID: "0f37b6a2-bdf4-4e07-b27c-e694d9576680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:21.466945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.466924 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8" pod "0f37b6a2-bdf4-4e07-b27c-e694d9576680" (UID: "0f37b6a2-bdf4-4e07-b27c-e694d9576680"). InnerVolumeSpecName "kube-api-access-vg4n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:21.532379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.532347 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt"] Oct 30 00:45:21.537201 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:21.537166 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24395531_dc30_4ca6_938f_1b153b905b19.slice/crio-a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439 WatchSource:0}: Error finding container a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439: Status 404 returned error can't find the container with id a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439 Oct 30 00:45:21.563657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.563633 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\") pod \"72bb7263-2ad1-474f-a932-b63edc372453\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " Oct 30 00:45:21.563754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.563717 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\" (UniqueName: \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\") pod \"72bb7263-2ad1-474f-a932-b63edc372453\" (UID: \"72bb7263-2ad1-474f-a932-b63edc372453\") " Oct 30 00:45:21.563973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.563950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca" pod "72bb7263-2ad1-474f-a932-b63edc372453" (UID: "72bb7263-2ad1-474f-a932-b63edc372453"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:21.564046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.564036 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4n8\" (UniqueName: \"kubernetes.io/projected/0f37b6a2-bdf4-4e07-b27c-e694d9576680-kube-api-access-vg4n8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:21.564083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.564049 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb7263-2ad1-474f-a932-b63edc372453-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:21.564083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.564060 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:21.564083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.564069 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f37b6a2-bdf4-4e07-b27c-e694d9576680-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:21.565500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.565480 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx" pod "72bb7263-2ad1-474f-a932-b63edc372453" (UID: "72bb7263-2ad1-474f-a932-b63edc372453"). InnerVolumeSpecName "kube-api-access-877hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:21.664946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.664924 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-877hx\" (UniqueName: \"kubernetes.io/projected/72bb7263-2ad1-474f-a932-b63edc372453-kube-api-access-877hx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:21.875378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:21.875352 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5895283d-4473-490e-aee9-fa029122306c" path="/var/lib/kubelet/pods/5895283d-4473-490e-aee9-fa029122306c/volumes" Oct 30 00:45:22.215301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.215274 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" event={"ID":"24395531-dc30-4ca6-938f-1b153b905b19","Type":"ContainerStarted","Data":"cb6022be04b6d4307608aed9b98eaebb9d03a4bb43f4320bbc3b7064cbaf2707"} Oct 30 00:45:22.215673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.215329 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" event={"ID":"24395531-dc30-4ca6-938f-1b153b905b19","Type":"ContainerStarted","Data":"a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439"} Oct 30 00:45:22.216531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.216513 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" Oct 30 00:45:22.216599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.216566 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p" event={"ID":"0f37b6a2-bdf4-4e07-b27c-e694d9576680","Type":"ContainerDied","Data":"1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334"} Oct 30 00:45:22.216654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.216604 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2be6b2a9740d23ae1c274ff266aa09d26adf697b924209e62c8277155a6334" Oct 30 00:45:22.217956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.217933 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" event={"ID":"72bb7263-2ad1-474f-a932-b63edc372453","Type":"ContainerDied","Data":"8ec249dba248c0673fb38db26b5f5ccbb90eab62f1de92280341a632f2efae7a"} Oct 30 00:45:22.218048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.217972 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b" Oct 30 00:45:22.218048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.217984 5938 scope.go:122] "RemoveContainer" containerID="49856d737661f718fe15d2702a16bd30717875ea978d73476551bfbf505068f2" Oct 30 00:45:22.257986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.257935 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" podStartSLOduration=2.257917505 podStartE2EDuration="2.257917505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:22.253904868 +0000 UTC m=+774.494139982" watchObservedRunningTime="2026-10-30 00:45:22.257917505 +0000 UTC m=+774.498152619" Oct 30 00:45:22.308203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.308176 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:22.310504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.310480 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888685-jmf2b"] Oct 30 00:45:22.515399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515377 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf"] Oct 30 00:45:22.515901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515884 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0f37b6a2-bdf4-4e07-b27c-e694d9576680" containerName="revoke-keys" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515903 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f37b6a2-bdf4-4e07-b27c-e694d9576680" containerName="revoke-keys" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515929 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515935 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515941 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515946 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515956 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.515965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515962 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515975 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.515979 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.516058 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f37b6a2-bdf4-4e07-b27c-e694d9576680" containerName="revoke-keys" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.516067 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.516072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.516078 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bb7263-2ad1-474f-a932-b63edc372453" containerName="cleanup" Oct 30 00:45:22.516165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.516086 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5895283d-4473-490e-aee9-fa029122306c" containerName="cleanup" Oct 30 00:45:22.519222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.519209 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.542070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.542045 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf"] Oct 30 00:45:22.577144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.577118 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.577275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.577159 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.577275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.577199 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hld8t\" (UniqueName: \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.678510 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.678484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hld8t\" (UniqueName: \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.678662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.678644 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.678746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.678699 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.679337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.679313 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.680842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.680822 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.701622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.701594 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hld8t\" (UniqueName: \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\") pod \"maas-api-revoke-sub-7d6eaa96-h29wf\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.851100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.851017 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:22.878021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:22.877996 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:23.154340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:23.154317 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf"] Oct 30 00:45:23.156352 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:23.156324 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978c87ae_fcc4_4324_b0f8_d27a03867812.slice/crio-941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df WatchSource:0}: Error finding container 941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df: Status 404 returned error can't find the container with id 941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df Oct 30 00:45:23.225910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:23.225879 5938 generic.go:356] "Generic (PLEG): container finished" podID="24395531-dc30-4ca6-938f-1b153b905b19" containerID="cb6022be04b6d4307608aed9b98eaebb9d03a4bb43f4320bbc3b7064cbaf2707" exitCode=0 Oct 30 00:45:23.226258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:23.225927 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" event={"ID":"24395531-dc30-4ca6-938f-1b153b905b19","Type":"ContainerDied","Data":"cb6022be04b6d4307608aed9b98eaebb9d03a4bb43f4320bbc3b7064cbaf2707"} Oct 30 00:45:23.227149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:23.227128 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" event={"ID":"978c87ae-fcc4-4324-b0f8-d27a03867812","Type":"ContainerStarted","Data":"941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df"} Oct 30 00:45:23.875074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:23.875039 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bb7263-2ad1-474f-a932-b63edc372453" path="/var/lib/kubelet/pods/72bb7263-2ad1-474f-a932-b63edc372453/volumes" Oct 30 00:45:24.235052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.234971 5938 generic.go:356] "Generic (PLEG): container finished" podID="978c87ae-fcc4-4324-b0f8-d27a03867812" containerID="fd6d885ef2db48152f88cc1a090c983177c921914591f710cd72cd1812de6d70" exitCode=0 Oct 30 00:45:24.235457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.235064 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" event={"ID":"978c87ae-fcc4-4324-b0f8-d27a03867812","Type":"ContainerDied","Data":"fd6d885ef2db48152f88cc1a090c983177c921914591f710cd72cd1812de6d70"} Oct 30 00:45:24.376973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.376950 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:24.498754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.498674 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\") pod \"24395531-dc30-4ca6-938f-1b153b905b19\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " Oct 30 00:45:24.498892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.498777 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\") pod \"24395531-dc30-4ca6-938f-1b153b905b19\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " Oct 30 00:45:24.499013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.498993 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\" (UniqueName: \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\") pod \"24395531-dc30-4ca6-938f-1b153b905b19\" (UID: \"24395531-dc30-4ca6-938f-1b153b905b19\") " Oct 30 00:45:24.499065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.499042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca" pod "24395531-dc30-4ca6-938f-1b153b905b19" (UID: "24395531-dc30-4ca6-938f-1b153b905b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:24.499478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.499435 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:24.501070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.501048 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr" pod "24395531-dc30-4ca6-938f-1b153b905b19" (UID: "24395531-dc30-4ca6-938f-1b153b905b19"). InnerVolumeSpecName "kube-api-access-7zstr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:24.501070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.501062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca" pod "24395531-dc30-4ca6-938f-1b153b905b19" (UID: "24395531-dc30-4ca6-938f-1b153b905b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:24.600798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.600767 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zstr\" (UniqueName: \"kubernetes.io/projected/24395531-dc30-4ca6-938f-1b153b905b19-kube-api-access-7zstr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:24.600798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.600798 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24395531-dc30-4ca6-938f-1b153b905b19-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:24.621277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:24.621252 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:25.244411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.244367 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" event={"ID":"24395531-dc30-4ca6-938f-1b153b905b19","Type":"ContainerDied","Data":"a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439"} Oct 30 00:45:25.244411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.244394 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt" Oct 30 00:45:25.244944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.244433 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c44b0f20e2edc6fd2498b7b931feda9db9a75394ea2e87d3b571990a2b5439" Oct 30 00:45:25.506991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.506639 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:25.587693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.587354 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\") pod \"978c87ae-fcc4-4324-b0f8-d27a03867812\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " Oct 30 00:45:25.587693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.587460 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\") pod \"978c87ae-fcc4-4324-b0f8-d27a03867812\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " Oct 30 00:45:25.587970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.587813 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca" pod "978c87ae-fcc4-4324-b0f8-d27a03867812" (UID: "978c87ae-fcc4-4324-b0f8-d27a03867812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:25.587970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.587902 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\" (UniqueName: \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\") pod \"978c87ae-fcc4-4324-b0f8-d27a03867812\" (UID: \"978c87ae-fcc4-4324-b0f8-d27a03867812\") " Oct 30 00:45:25.591882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.588311 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:25.592173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.592132 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca" pod "978c87ae-fcc4-4324-b0f8-d27a03867812" (UID: "978c87ae-fcc4-4324-b0f8-d27a03867812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:25.596816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.596792 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t" pod "978c87ae-fcc4-4324-b0f8-d27a03867812" (UID: "978c87ae-fcc4-4324-b0f8-d27a03867812"). InnerVolumeSpecName "kube-api-access-hld8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:25.694994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.690099 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978c87ae-fcc4-4324-b0f8-d27a03867812-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:25.694994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:25.690136 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hld8t\" (UniqueName: \"kubernetes.io/projected/978c87ae-fcc4-4324-b0f8-d27a03867812-kube-api-access-hld8t\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:26.263118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:26.263000 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" event={"ID":"978c87ae-fcc4-4324-b0f8-d27a03867812","Type":"ContainerDied","Data":"941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df"} Oct 30 00:45:26.263722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:26.263690 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941a040be15a89610af65dfadf8592b0ba317003dba8a0ad895e42fd60d429df" Oct 30 00:45:26.263985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:26.263968 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf" Oct 30 00:45:26.456932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:26.456897 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:28.466617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:28.466581 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:28.834356 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:28.834273 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:45:30.056347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:30.056315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:31.070646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.070611 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj"] Oct 30 00:45:31.071099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071084 5938 cpu_manager.go:414] "containerMap: removing container" podUID="24395531-dc30-4ca6-938f-1b153b905b19" containerName="revoke-keys" Oct 30 00:45:31.071099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071099 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24395531-dc30-4ca6-938f-1b153b905b19" containerName="revoke-keys" Oct 30 00:45:31.071184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071117 5938 cpu_manager.go:414] "containerMap: removing container" podUID="978c87ae-fcc4-4324-b0f8-d27a03867812" containerName="revoke-keys" Oct 30 00:45:31.071184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071122 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978c87ae-fcc4-4324-b0f8-d27a03867812" containerName="revoke-keys" Oct 30 00:45:31.071245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071197 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="24395531-dc30-4ca6-938f-1b153b905b19" containerName="revoke-keys" Oct 30 00:45:31.071245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.071207 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="978c87ae-fcc4-4324-b0f8-d27a03867812" containerName="revoke-keys" Oct 30 00:45:31.075905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.075889 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.092000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.091966 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj"] Oct 30 00:45:31.100717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.100288 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:45:31.100717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.100315 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:45:31.100717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.100341 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:45:31.102031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.102013 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:45:31.192637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.192612 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.192769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.192666 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.192853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.192826 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmj2\" (UniqueName: \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.294189 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.294157 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmj2\" (UniqueName: \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.294311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.294292 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.294402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.294344 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.295044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.295014 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.296776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.296755 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.316457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.316432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmj2\" (UniqueName: \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\") pod \"maas-api-revoke-sub-0c9ce1bd-54knj\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.388650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.388630 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:31.408904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.408877 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 00:45:31.417188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.417156 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.431898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.431444 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca5363\"/\"distinct-w4-ca5363-kserve-self-signed-certs\"" Oct 30 00:45:31.433117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.432834 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 00:45:31.518364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.518337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:31.597289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597263 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.597411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597311 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.597411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597330 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjqd\" (UniqueName: \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.597484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597420 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.597484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597458 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.597607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.597585 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699173 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699229 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699251 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjqd\" (UniqueName: \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699318 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699389 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699643 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj"] Oct 30 00:45:31.699764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699713 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.699958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.699927 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.701527 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:31.701503 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511bfec0_5b99_4c6a_a9bc_da6582179cbb.slice/crio-ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d WatchSource:0}: Error finding container ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d: Status 404 returned error can't find the container with id ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d Oct 30 00:45:31.701631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.701603 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.701894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.701856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.721564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.721543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjqd\" (UniqueName: \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\") pod \"distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.733062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.733043 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:31.896015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:31.895983 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:32.034182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.034153 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 00:45:32.035609 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:32.035584 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641fb22e_6646_4270_a40c_8515fe2f8e4b.slice/crio-03355d3851a4aaea1e0c3028c32b93ce9a15f7d49538497d4aad2f0d19591168 WatchSource:0}: Error finding container 03355d3851a4aaea1e0c3028c32b93ce9a15f7d49538497d4aad2f0d19591168: Status 404 returned error can't find the container with id 03355d3851a4aaea1e0c3028c32b93ce9a15f7d49538497d4aad2f0d19591168 Oct 30 00:45:32.305951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.305837 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" event={"ID":"641fb22e-6646-4270-a40c-8515fe2f8e4b","Type":"ContainerStarted","Data":"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817"} Oct 30 00:45:32.305951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.305901 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" event={"ID":"641fb22e-6646-4270-a40c-8515fe2f8e4b","Type":"ContainerStarted","Data":"03355d3851a4aaea1e0c3028c32b93ce9a15f7d49538497d4aad2f0d19591168"} Oct 30 00:45:32.307907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.307856 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" event={"ID":"511bfec0-5b99-4c6a-a9bc-da6582179cbb","Type":"ContainerStarted","Data":"191f0166b723c2f50e9ea3a6263661640032c64056d240889575048cb0f4f3b2"} Oct 30 00:45:32.308006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.307958 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" event={"ID":"511bfec0-5b99-4c6a-a9bc-da6582179cbb","Type":"ContainerStarted","Data":"ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d"} Oct 30 00:45:32.347331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.347283 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podStartSLOduration=1.347270339 podStartE2EDuration="1.347270339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:32.345486778 +0000 UTC m=+784.585721890" watchObservedRunningTime="2026-10-30 00:45:32.347270339 +0000 UTC m=+784.587505525" Oct 30 00:45:32.386564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.386518 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" podStartSLOduration=1.386506896 podStartE2EDuration="1.386506896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:32.383920026 +0000 UTC m=+784.624155140" watchObservedRunningTime="2026-10-30 00:45:32.386506896 +0000 UTC m=+784.626742041" Oct 30 00:45:32.734308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:32.734281 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:33.334075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:33.334040 5938 generic.go:356] "Generic (PLEG): container finished" podID="511bfec0-5b99-4c6a-a9bc-da6582179cbb" containerID="191f0166b723c2f50e9ea3a6263661640032c64056d240889575048cb0f4f3b2" exitCode=0 Oct 30 00:45:33.340886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:33.335625 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" event={"ID":"511bfec0-5b99-4c6a-a9bc-da6582179cbb","Type":"ContainerDied","Data":"191f0166b723c2f50e9ea3a6263661640032c64056d240889575048cb0f4f3b2"} Oct 30 00:45:33.415739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:33.415699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:34.484814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.484785 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:34.551059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.551032 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\") pod \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " Oct 30 00:45:34.551215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.551128 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\") pod \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " Oct 30 00:45:34.551215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.551177 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\" (UniqueName: \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\") pod \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\" (UID: \"511bfec0-5b99-4c6a-a9bc-da6582179cbb\") " Oct 30 00:45:34.551422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.551397 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca" pod "511bfec0-5b99-4c6a-a9bc-da6582179cbb" (UID: "511bfec0-5b99-4c6a-a9bc-da6582179cbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:34.551601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.551571 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:34.553218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.553197 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca" pod "511bfec0-5b99-4c6a-a9bc-da6582179cbb" (UID: "511bfec0-5b99-4c6a-a9bc-da6582179cbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:34.553297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.553228 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2" pod "511bfec0-5b99-4c6a-a9bc-da6582179cbb" (UID: "511bfec0-5b99-4c6a-a9bc-da6582179cbb"). InnerVolumeSpecName "kube-api-access-6hmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:34.652466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.652442 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511bfec0-5b99-4c6a-a9bc-da6582179cbb-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:34.652466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.652466 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmj2\" (UniqueName: \"kubernetes.io/projected/511bfec0-5b99-4c6a-a9bc-da6582179cbb-kube-api-access-6hmj2\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:34.982476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:34.982404 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:35.348557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.348488 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" Oct 30 00:45:35.348687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.348547 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj" event={"ID":"511bfec0-5b99-4c6a-a9bc-da6582179cbb","Type":"ContainerDied","Data":"ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d"} Oct 30 00:45:35.348687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.348597 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8ce6d6511fc77f41502bb9581cde106bab31eb4c604db51c81872aafeb1b7d" Oct 30 00:45:35.419242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.419216 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:45:35.419687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.419676 5938 cpu_manager.go:414] "containerMap: removing container" podUID="511bfec0-5b99-4c6a-a9bc-da6582179cbb" containerName="revoke-keys" Oct 30 00:45:35.419738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.419689 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511bfec0-5b99-4c6a-a9bc-da6582179cbb" containerName="revoke-keys" Oct 30 00:45:35.419779 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.419770 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="511bfec0-5b99-4c6a-a9bc-da6582179cbb" containerName="revoke-keys" Oct 30 00:45:35.423457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.423438 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.447092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.447067 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:45:35.462941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.462910 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.462981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463035 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463151 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463358 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463401 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nbg\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463481 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.463542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.463531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.517542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.517509 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 00:45:35.521481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.521461 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.554091 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.554062 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 00:45:35.560152 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.560126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:35.564409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564385 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7tf\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564472 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564503 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nbg\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564552 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564576 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564601 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564635 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564657 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564673 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564710 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564740 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564760 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564791 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.564840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564823 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.565742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564905 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.565742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.564966 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.565742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.565000 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.565742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.565197 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.565742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.565307 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.566020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.565747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.566648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.566606 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.568660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.568636 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.570098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.570074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.587620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.587598 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nbg\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.587814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.587793 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.666009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.665980 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666085 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7tf\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666184 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666224 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666280 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666344 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666388 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666766 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.666794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666786 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.667031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666840 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.667031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.666896 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.667031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.667010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.668522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.668499 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.669354 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.669324 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.689264 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.689240 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.689431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.689415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7tf\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:35.736351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.736324 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:35.853712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:35.853674 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:36.053682 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.053659 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:45:36.054879 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:36.054837 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213e2306_e63e_478a_9d7d_c1c5038343e3.slice/crio-3f0e45795fae9b05bc319888f8049a276d84a2011d75c0b96c11bdbafc43913e WatchSource:0}: Error finding container 3f0e45795fae9b05bc319888f8049a276d84a2011d75c0b96c11bdbafc43913e: Status 404 returned error can't find the container with id 3f0e45795fae9b05bc319888f8049a276d84a2011d75c0b96c11bdbafc43913e Oct 30 00:45:36.057237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.057205 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.057301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.057274 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.057338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.057306 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.161766 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.161725 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 00:45:36.161966 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:36.161930 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d3df76_2e83_418a_ae74_51ef603bca7f.slice/crio-fde88e0f80d65db6b53ebf738551a5b36ce1386451c34bfaab57877ec205a060 WatchSource:0}: Error finding container fde88e0f80d65db6b53ebf738551a5b36ce1386451c34bfaab57877ec205a060: Status 404 returned error can't find the container with id fde88e0f80d65db6b53ebf738551a5b36ce1386451c34bfaab57877ec205a060 Oct 30 00:45:36.164988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.164949 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.165061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.165037 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.165107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.165082 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:36.359089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.359055 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" event={"ID":"213e2306-e63e-478a-9d7d-c1c5038343e3","Type":"ContainerStarted","Data":"ec254a66005b5205c80073a5a48af851e19c2067e551e35aed43a7674ca162c4"} Oct 30 00:45:36.359335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.359317 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" event={"ID":"213e2306-e63e-478a-9d7d-c1c5038343e3","Type":"ContainerStarted","Data":"3f0e45795fae9b05bc319888f8049a276d84a2011d75c0b96c11bdbafc43913e"} Oct 30 00:45:36.363029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.362999 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" event={"ID":"d1d3df76-2e83-418a-ae74-51ef603bca7f","Type":"ContainerStarted","Data":"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f"} Oct 30 00:45:36.365899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.363050 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" event={"ID":"d1d3df76-2e83-418a-ae74-51ef603bca7f","Type":"ContainerStarted","Data":"fde88e0f80d65db6b53ebf738551a5b36ce1386451c34bfaab57877ec205a060"} Oct 30 00:45:36.402032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.401979 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podStartSLOduration=1.401962549 podStartE2EDuration="1.401962549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:36.400916393 +0000 UTC m=+788.641151505" watchObservedRunningTime="2026-10-30 00:45:36.401962549 +0000 UTC m=+788.642197667" Oct 30 00:45:36.444372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.444323 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podStartSLOduration=1.444307056 podStartE2EDuration="1.444307056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:36.443498894 +0000 UTC m=+788.683734008" watchObservedRunningTime="2026-10-30 00:45:36.444307056 +0000 UTC m=+788.684542172" Oct 30 00:45:36.671296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.671215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:36.738462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.738362 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:36.851491 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.851461 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:36.854591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:36.854568 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:37.739939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:37.738939 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:37.739939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:37.739012 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:37.855347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:37.855316 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:37.855514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:37.855365 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:38.737113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:38.737078 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:38.737387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:38.737360 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:38.817873 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:38.817814 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:38.854386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:38.854353 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:38.854553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:38.854428 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:39.739018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:39.738984 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:39.739302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:39.739271 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:39.854407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:39.854371 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:39.855074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:39.855041 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:40.583887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.577951 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:40.620887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.613967 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:45:40.738048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.738003 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:40.738893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.738541 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:40.739774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.739748 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:40.788769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.788728 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:40.855345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.855314 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:40.855924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.855893 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:40.889957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:40.889919 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:45:41.267961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.267038 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:41.482985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.482940 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:41.579140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.579016 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:41.579911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.579858 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:45:41.668001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.667952 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:41.753896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.743299 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:45:41.753896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.743804 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:45:41.753896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.746521 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:41.803887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.800818 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:45:41.803887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.801330 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:45:41.901260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.895963 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:41.901260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:41.896436 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:45:42.004909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.004290 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:42.004909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.004341 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:42.021493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.005368 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:42.021493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.005401 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:42.240012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.225803 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:42.283681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.269766 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:45:42.283681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.270318 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:45:42.283681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.270758 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:42.283681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.271084 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:45:42.321916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.315755 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:45:42.321916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.315874 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:42.447026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.442277 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:42.666155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.666118 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:45:42.671966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.671923 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:42.681880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.672557 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:45:42.755973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.753963 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:42.780046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.780006 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:42.780285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.780263 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:42.930977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.894475 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:45:42.958047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.958018 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:45:42.962027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.917323 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:42.968975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:42.968938 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:43.232458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.231945 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:43.291885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.275635 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:45:43.415884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.389427 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:45:43.734217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.734191 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:45:43.738259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.738229 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 00:45:43.738468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.738441 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 00:45:43.857612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.857573 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c"] Oct 30 00:45:43.861239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.861212 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 00:45:43.861485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.861455 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 00:45:43.882911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.867274 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:43.883780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.883743 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:45:43.917208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.910195 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:45:43.917208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.910268 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:45:43.917208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.910366 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:45:43.947240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.919777 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:43.947240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.919959 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:43.947240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.920029 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsh9\" (UniqueName: \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:43.999001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:43.998549 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c"] Oct 30 00:45:44.021710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.021665 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.021938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.021795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.021938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.021839 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsh9\" (UniqueName: \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.022625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.022602 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.024671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.024645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.045290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.045254 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsh9\" (UniqueName: \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\") pod \"maas-api-revoke-sub-1a45792a-jmj4c\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.086791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.086729 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:44.270995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.270904 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:44.593029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:44.593005 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c"] Oct 30 00:45:44.595922 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:45:44.595892 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed01342_41b2_4c67_b09e_a05dc6c10ba4.slice/crio-1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7 WatchSource:0}: Error finding container 1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7: Status 404 returned error can't find the container with id 1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7 Oct 30 00:45:45.214613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.214528 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:45.215801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.215780 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:45.216768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.216746 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:45:45.519416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.519383 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:45.520138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.520006 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:45.521642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.521083 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 00:45:45.552662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.552628 5938 generic.go:356] "Generic (PLEG): container finished" podID="6ed01342-41b2-4c67-b09e-a05dc6c10ba4" containerID="0e0b57c12533673f77ea3ce120d2554d56957e91eb018c29295bb68508b29b97" exitCode=0 Oct 30 00:45:45.552768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.552675 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" event={"ID":"6ed01342-41b2-4c67-b09e-a05dc6c10ba4","Type":"ContainerDied","Data":"0e0b57c12533673f77ea3ce120d2554d56957e91eb018c29295bb68508b29b97"} Oct 30 00:45:45.552768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.552727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" event={"ID":"6ed01342-41b2-4c67-b09e-a05dc6c10ba4","Type":"ContainerStarted","Data":"1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7"} Oct 30 00:45:45.806195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:45.806104 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:46.704237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.704213 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:46.766175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.766147 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\") pod \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " Oct 30 00:45:46.766316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.766255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\" (UniqueName: \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\") pod \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " Oct 30 00:45:46.766384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.766344 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\") pod \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\" (UID: \"6ed01342-41b2-4c67-b09e-a05dc6c10ba4\") " Oct 30 00:45:46.766662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.766636 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca" pod "6ed01342-41b2-4c67-b09e-a05dc6c10ba4" (UID: "6ed01342-41b2-4c67-b09e-a05dc6c10ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:46.766801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.766779 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:46.768330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.768300 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca" pod "6ed01342-41b2-4c67-b09e-a05dc6c10ba4" (UID: "6ed01342-41b2-4c67-b09e-a05dc6c10ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:46.768662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.768643 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9" pod "6ed01342-41b2-4c67-b09e-a05dc6c10ba4" (UID: "6ed01342-41b2-4c67-b09e-a05dc6c10ba4"). InnerVolumeSpecName "kube-api-access-6dsh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:46.867599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.867576 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:46.867599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:46.867599 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsh9\" (UniqueName: \"kubernetes.io/projected/6ed01342-41b2-4c67-b09e-a05dc6c10ba4-kube-api-access-6dsh9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:45:47.342302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:47.342207 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:47.567361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:47.567331 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" event={"ID":"6ed01342-41b2-4c67-b09e-a05dc6c10ba4","Type":"ContainerDied","Data":"1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7"} Oct 30 00:45:47.567361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:47.567352 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c" Oct 30 00:45:47.567557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:47.567378 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd7d2627e7bd0b692f1e45da2a58f8b4341ab6dc9a86fc916c77a260ddebbc7" Oct 30 00:45:49.073890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:49.073838 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:50.577620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.577587 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:50.642139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.642103 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:45:50.642677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.642658 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6ed01342-41b2-4c67-b09e-a05dc6c10ba4" containerName="revoke-keys" Oct 30 00:45:50.642729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.642677 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed01342-41b2-4c67-b09e-a05dc6c10ba4" containerName="revoke-keys" Oct 30 00:45:50.642798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.642788 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed01342-41b2-4c67-b09e-a05dc6c10ba4" containerName="revoke-keys" Oct 30 00:45:50.647341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.647322 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.661606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.661582 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-52650f-openshift-default-dockercfg-5tktc\"" Oct 30 00:45:50.676393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.676370 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:45:50.810852 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.810823 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.810995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.810938 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.810995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.810964 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811038 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811068 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811104 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811231 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p565q\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811281 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.811402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.811386 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.911821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911798 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.911956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911839 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.911956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911885 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.911956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p565q\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.911956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911931 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911962 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.911998 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912071 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912103 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912448 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912521 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912616 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.912798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.912769 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.914393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.914364 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.914481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.914438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.934238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.934209 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.934419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.934400 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p565q\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:50.959756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:50.959732 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:51.177904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.177810 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:51.289943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.289768 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:45:51.313197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.312298 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:51.313197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.312424 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:51.313197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.312469 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:45:51.595816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.595739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" event={"ID":"50729a04-4c87-429d-8354-4338ac1197e1","Type":"ContainerStarted","Data":"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a"} Oct 30 00:45:51.595816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.595798 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" event={"ID":"50729a04-4c87-429d-8354-4338ac1197e1","Type":"ContainerStarted","Data":"3e24f90079921ba104fd938e109f58e486525839a8e9bdac17e238a051e0409d"} Oct 30 00:45:51.641817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.641755 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podStartSLOduration=1.6417391700000001 podStartE2EDuration="1.64173917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:51.640007606 +0000 UTC m=+803.880242724" watchObservedRunningTime="2026-10-30 00:45:51.64173917 +0000 UTC m=+803.881974286" Oct 30 00:45:51.960153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:51.960121 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:52.265056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:52.264973 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:52.961800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:52.961764 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:52.962276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:52.961821 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:53.775945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:53.775917 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:53.960978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:53.960941 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:53.961166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:53.961002 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:54.182439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:54.182405 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:54.960251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:54.960214 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:54.960420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:54.960262 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:55.571019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:55.570982 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:55.960412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:55.960383 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:55.960646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:55.960439 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:56.960138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:56.960105 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:56.960541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:56.960172 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:57.082901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:57.081672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:57.960571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:57.960539 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 00:45:57.960970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:57.960595 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 00:45:58.592299 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:58.592266 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:58.833736 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:45:58.833699 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:45:58.964273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:58.964244 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:58.964660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:58.964642 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:58.965347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:58.965328 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:45:59.893555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.893523 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:45:59.898715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.898692 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.929406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.929379 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:45:59.932050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.932025 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:45:59.935986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.935969 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.957085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.957058 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:45:59.996929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.996902 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.996935 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.996959 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997028 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997107 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997216 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslsl\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997282 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997389 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997424 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997489 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997519 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997549 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997598 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997630 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997653 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvcz\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:45:59.997820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997673 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:45:59.997820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:45:59.997733 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098480 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098455 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098512 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098540 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098652 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslsl\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098686 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098748 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098783 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.098910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098837 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.098910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098885 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098915 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098959 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.098968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099027 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099063 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvcz\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099095 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.099481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099135 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.099481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099146 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.099481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.099438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.100074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100050 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.100219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100060 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.100324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100157 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.100466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100439 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.100595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.100786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.100702 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.101421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.101400 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.101791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.101763 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.101791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.101778 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.103024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.103000 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.121571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.121540 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.121658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.121593 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.121658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.121605 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslsl\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\") pod \"maas-default-gateway-openshift-default-7d9897b54c-s47sp\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.122247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.122223 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvcz\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\") pod \"maas-default-gateway-openshift-default-7d9897b54c-gfvgn\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.213537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.213480 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:00.234457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.234376 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:00.248598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.248570 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:00.546828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.546799 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:46:00.548226 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:00.548199 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4337f8_db9f_4a1b_b47a_8839844fe021.slice/crio-0cc651565da0ea1e941c0c60f441f556876816db01428549376d84d0f7e380c2 WatchSource:0}: Error finding container 0cc651565da0ea1e941c0c60f441f556876816db01428549376d84d0f7e380c2: Status 404 returned error can't find the container with id 0cc651565da0ea1e941c0c60f441f556876816db01428549376d84d0f7e380c2 Oct 30 00:46:00.550576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.550546 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.550640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.550609 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.550677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.550639 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.586916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.586892 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:46:00.593912 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:00.593885 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70d7096_5107_48f2_8732_34d17da6a020.slice/crio-6fc1d8d347cf0519d77b652e54f61676b3c7b0e9c419a71c057bf46f0786df36 WatchSource:0}: Error finding container 6fc1d8d347cf0519d77b652e54f61676b3c7b0e9c419a71c057bf46f0786df36: Status 404 returned error can't find the container with id 6fc1d8d347cf0519d77b652e54f61676b3c7b0e9c419a71c057bf46f0786df36 Oct 30 00:46:00.596547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.596515 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.596634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.596581 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.596634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.596614 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:00.699885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.696894 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" event={"ID":"f70d7096-5107-48f2-8732-34d17da6a020","Type":"ContainerStarted","Data":"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b"} Oct 30 00:46:00.699885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.696958 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" event={"ID":"f70d7096-5107-48f2-8732-34d17da6a020","Type":"ContainerStarted","Data":"6fc1d8d347cf0519d77b652e54f61676b3c7b0e9c419a71c057bf46f0786df36"} Oct 30 00:46:00.699885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.699021 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" event={"ID":"cd4337f8-db9f-4a1b-b47a-8839844fe021","Type":"ContainerStarted","Data":"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd"} Oct 30 00:46:00.699885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.699061 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" event={"ID":"cd4337f8-db9f-4a1b-b47a-8839844fe021","Type":"ContainerStarted","Data":"0cc651565da0ea1e941c0c60f441f556876816db01428549376d84d0f7e380c2"} Oct 30 00:46:00.746734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.746687 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podStartSLOduration=1.746673481 podStartE2EDuration="1.746673481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:00.741441202 +0000 UTC m=+812.981676316" watchObservedRunningTime="2026-10-30 00:46:00.746673481 +0000 UTC m=+812.986908591" Oct 30 00:46:00.788219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:00.788168 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podStartSLOduration=1.788153704 podStartE2EDuration="1.788153704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:00.784591008 +0000 UTC m=+813.024826153" watchObservedRunningTime="2026-10-30 00:46:00.788153704 +0000 UTC m=+813.028388816" Oct 30 00:46:01.213726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.213687 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:01.215298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.215274 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:01.215406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.215326 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:01.251935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.251896 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:01.255229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.254991 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:01.255229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.255066 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:01.861189 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:01.861116 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:02.217363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:02.217274 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:02.217972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:02.217936 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:02.249020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:02.248987 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:02.249220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:02.249200 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:03.224926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:03.221254 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:03.224926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:03.221334 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:03.250237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:03.250134 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:03.250237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:03.250190 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:03.612889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:03.606616 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:04.223892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.222342 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:04.223892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.222401 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:04.249890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.249106 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:04.249890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.249160 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:04.486945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.485949 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:04.490886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.487158 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:04.608889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.607952 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:04.670974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.670927 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:46:04.747037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.746938 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:04.765153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.765115 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:46:04.963031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:04.962982 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:05.175887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.174909 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:05.264890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.259623 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:05.264890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.259679 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:05.264890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.261356 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:05.264890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.261396 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:05.264890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.261578 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:46:05.273521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.273469 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:46:05.352883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.349886 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:46:05.352883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.350010 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473093 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473140 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28bp\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473368 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473541 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473583 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473646 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473771 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473793 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.473933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.473808 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.524971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.524923 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:05.574657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574619 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.574822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574667 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.574822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.574822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574749 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.574822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574781 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28bp\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574844 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.574982 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575009 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575140 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575194 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575235 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575704 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.575941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.575920 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.577607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.577556 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.578019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.578001 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.615892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.611438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28bp\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.639887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.626367 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:05.744055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:05.743955 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:06.143341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.142730 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:46:06.150668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.150626 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:06.150789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.150701 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:06.150789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.150737 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:06.214255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.214224 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:06.214443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.214416 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:06.249145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.249122 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:06.249329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.249306 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:06.301915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.300986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:06.849584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.849548 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" event={"ID":"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3","Type":"ContainerStarted","Data":"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08"} Oct 30 00:46:06.849850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.849602 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" event={"ID":"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3","Type":"ContainerStarted","Data":"0caa808f847f429e9611501f523a2e5bc122a6461ef1170b474da9faa6ac6a6e"} Oct 30 00:46:06.915886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:06.906737 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podStartSLOduration=1.906716402 podStartE2EDuration="1.906716402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:06.903153442 +0000 UTC m=+819.143388557" watchObservedRunningTime="2026-10-30 00:46:06.906716402 +0000 UTC m=+819.146951515" Oct 30 00:46:07.182824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.182789 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:07.232634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.218346 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:07.232634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.218422 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:07.256892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.256226 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:07.256892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.256286 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:07.745121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.745080 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:07.746820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.746786 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:07.746991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:07.746840 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:08.219885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.219054 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 00:46:08.219885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.219131 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 00:46:08.249624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.249595 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:08.249784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.249641 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:08.746347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.746314 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:08.746736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.746365 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:08.875879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:08.874363 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:09.025972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:09.025743 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:09.250490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:09.250453 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-gfvgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 00:46:09.251009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:09.250967 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 00:46:09.744796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:09.744751 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:09.745129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:09.745088 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:10.268046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.267995 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-s47sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:46:10.268525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.268500 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:46:10.531699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.531606 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:10.532125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.532103 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:10.533466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.533445 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:46:10.574395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.574292 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:10.721265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.721231 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:10.723991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.723966 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:10.725241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.725223 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:46:10.744955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.744886 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:10.745084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:10.744939 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:11.744776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:11.744741 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:11.745192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:11.744794 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:12.074761 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:12.074676 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:12.744637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:12.744605 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 00:46:12.744817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:12.744654 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 00:46:13.748667 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:13.748638 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:13.749089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:13.749037 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:13.749805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:13.749789 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:46:13.796053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:13.796020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:15.344386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:15.343374 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:17.074024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:17.073980 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:18.662306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:18.662267 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:19.775699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.775664 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 00:46:19.780518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.780494 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.796693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.796670 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 00:46:19.805043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.805012 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"distinct-w6-021d6d-kserve-self-signed-certs\"" Oct 30 00:46:19.832777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.832750 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.832916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.832793 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.832974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.832948 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8z9\" (UniqueName: \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.833012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.832975 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.833012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.833008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.833085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.833054 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.860686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.860655 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:19.933776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.933756 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8z9\" (UniqueName: \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.933923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.933787 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.933923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.933814 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.933923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.933833 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.934169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.934151 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.934237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.934196 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.934237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.934212 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.934343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.934269 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.934458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.934438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.936305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.936282 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.936442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.936426 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:19.999350 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:19.999304 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8z9\" (UniqueName: \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\") pod \"distinct-w6-021d6d-kserve-d97574bd6-cbb79\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:20.094112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.094037 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:20.344661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.344576 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:20.400162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.400141 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 00:46:20.401706 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:20.401672 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8094870f_eac3_4eae_922e_95a27b456eba.slice/crio-0bf1b514797f31275f8bbf80c8fca3c57161c3eb684b85728d9abc8e21ff2671 WatchSource:0}: Error finding container 0bf1b514797f31275f8bbf80c8fca3c57161c3eb684b85728d9abc8e21ff2671: Status 404 returned error can't find the container with id 0bf1b514797f31275f8bbf80c8fca3c57161c3eb684b85728d9abc8e21ff2671 Oct 30 00:46:20.558654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.558626 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:46:20.562750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.562728 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.586413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.586386 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:46:20.642206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642180 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642238 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642364 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642508 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642560 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sjc\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642587 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642656 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.642703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.642676 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743586 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743610 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sjc\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743873 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743738 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743873 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743819 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.743985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743857 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.743987 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744000 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744135 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744175 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744460 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.744543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.744485 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.746081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.746058 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.746561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.746535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.765747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.765724 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sjc\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.765846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.765819 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:20.877839 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:20.877819 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:21.008742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.008705 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" event={"ID":"8094870f-eac3-4eae-922e-95a27b456eba","Type":"ContainerStarted","Data":"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e"} Oct 30 00:46:21.008901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.008757 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" event={"ID":"8094870f-eac3-4eae-922e-95a27b456eba","Type":"ContainerStarted","Data":"0bf1b514797f31275f8bbf80c8fca3c57161c3eb684b85728d9abc8e21ff2671"} Oct 30 00:46:21.050912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.050854 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podStartSLOduration=2.050841192 podStartE2EDuration="2.050841192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:21.049616543 +0000 UTC m=+833.289851655" watchObservedRunningTime="2026-10-30 00:46:21.050841192 +0000 UTC m=+833.291076305" Oct 30 00:46:21.094101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.094080 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:21.196155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.195289 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:46:21.197380 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:21.197349 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34feee0_f121_4b7a_964a_aea2f35c1475.slice/crio-b1ddaf1a559fd551f3021d1dec04f9d003a20c2ffff5d44885df47971f3f2cdc WatchSource:0}: Error finding container b1ddaf1a559fd551f3021d1dec04f9d003a20c2ffff5d44885df47971f3f2cdc: Status 404 returned error can't find the container with id b1ddaf1a559fd551f3021d1dec04f9d003a20c2ffff5d44885df47971f3f2cdc Oct 30 00:46:21.200459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.200419 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:21.200564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.200508 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:21.200564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.200557 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:21.910377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:21.910339 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:22.032149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.032108 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" event={"ID":"a34feee0-f121-4b7a-964a-aea2f35c1475","Type":"ContainerStarted","Data":"25879a21ed10f6c99fbb02b655325f44f3dc0f6556ba957b02b0577fa4ad124f"} Oct 30 00:46:22.032329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.032158 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" event={"ID":"a34feee0-f121-4b7a-964a-aea2f35c1475","Type":"ContainerStarted","Data":"b1ddaf1a559fd551f3021d1dec04f9d003a20c2ffff5d44885df47971f3f2cdc"} Oct 30 00:46:22.121682 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.121628 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podStartSLOduration=2.121613295 podStartE2EDuration="2.121613295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:22.108410503 +0000 UTC m=+834.348645618" watchObservedRunningTime="2026-10-30 00:46:22.121613295 +0000 UTC m=+834.361848407" Oct 30 00:46:22.878090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.878054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:22.879605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.879575 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:46:22.879769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:22.879619 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 00:46:23.710108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:23.710070 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:23.878381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:23.878346 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:46:23.878553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:23.878406 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 00:46:24.878812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:24.878779 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:46:24.879249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:24.878849 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 00:46:25.713200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:25.713163 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:25.879143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:25.879108 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:46:25.879550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:25.879165 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 00:46:26.109541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:26.109506 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:26.878583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:26.878554 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 00:46:26.878789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:26.878600 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 00:46:27.660697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:27.660660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:27.882235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:27.882209 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:27.882579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:27.882557 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:27.883327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:27.883309 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:46:28.834971 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:46:28.834921 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:46:29.246360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.246327 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4"] Oct 30 00:46:29.252512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.252489 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.267659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.267637 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4"] Oct 30 00:46:29.274546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.274522 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:46:29.274685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.274629 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:46:29.274746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.274699 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:46:29.274746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.274726 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:46:29.311804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.311774 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:29.374978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.374944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49tz\" (UniqueName: \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.375191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.375094 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.375279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.375256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.476452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.476421 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.476615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.476525 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49tz\" (UniqueName: \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.476615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.476581 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.477160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.477134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.478967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.478945 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.499012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.498955 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49tz\" (UniqueName: \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\") pod \"maas-api-revoke-sub-e2b9bc9e-grrn4\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.574050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.574031 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:29.877936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:29.877911 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4"] Oct 30 00:46:29.878712 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:29.878687 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a95512_5fd7_4f30_9ef0_b495ed6a437c.slice/crio-67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b WatchSource:0}: Error finding container 67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b: Status 404 returned error can't find the container with id 67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b Oct 30 00:46:30.083825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:30.083796 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" event={"ID":"c4a95512-5fd7-4f30-9ef0-b495ed6a437c","Type":"ContainerStarted","Data":"67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b"} Oct 30 00:46:30.094142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:30.094112 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:30.107074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:30.107054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:30.910065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:30.910037 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:31.093183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:31.093095 5938 generic.go:356] "Generic (PLEG): container finished" podID="c4a95512-5fd7-4f30-9ef0-b495ed6a437c" containerID="f2b0cb2cc9adbbc84d214af5780988465557ca9d72e0710adffcefa5bcdf055f" exitCode=0 Oct 30 00:46:31.093183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:31.093170 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" event={"ID":"c4a95512-5fd7-4f30-9ef0-b495ed6a437c","Type":"ContainerDied","Data":"f2b0cb2cc9adbbc84d214af5780988465557ca9d72e0710adffcefa5bcdf055f"} Oct 30 00:46:31.104481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:31.104455 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:46:32.257262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.257238 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:32.309045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.309020 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\" (UniqueName: \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\") pod \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " Oct 30 00:46:32.309197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.309169 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\") pod \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " Oct 30 00:46:32.309274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.309239 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\") pod \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\" (UID: \"c4a95512-5fd7-4f30-9ef0-b495ed6a437c\") " Oct 30 00:46:32.309557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.309532 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca" pod "c4a95512-5fd7-4f30-9ef0-b495ed6a437c" (UID: "c4a95512-5fd7-4f30-9ef0-b495ed6a437c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:46:32.309809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.309787 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:32.311271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.311243 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz" pod "c4a95512-5fd7-4f30-9ef0-b495ed6a437c" (UID: "c4a95512-5fd7-4f30-9ef0-b495ed6a437c"). InnerVolumeSpecName "kube-api-access-r49tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:32.311436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.311417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca" pod "c4a95512-5fd7-4f30-9ef0-b495ed6a437c" (UID: "c4a95512-5fd7-4f30-9ef0-b495ed6a437c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:46:32.411587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.411553 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49tz\" (UniqueName: \"kubernetes.io/projected/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-kube-api-access-r49tz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:32.411587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.411588 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a95512-5fd7-4f30-9ef0-b495ed6a437c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:32.481205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:32.481174 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:33.107419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:33.107389 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" Oct 30 00:46:33.107592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:33.107389 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4" event={"ID":"c4a95512-5fd7-4f30-9ef0-b495ed6a437c","Type":"ContainerDied","Data":"67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b"} Oct 30 00:46:33.107592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:33.107515 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67742256b005fe3f194498764178e37544a559f33ec6b9b2e42320da5374936b" Oct 30 00:46:33.927157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:33.927113 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:35.402330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.402288 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:46:35.444902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.403708 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c4a95512-5fd7-4f30-9ef0-b495ed6a437c" containerName="revoke-keys" Oct 30 00:46:35.445155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.445129 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a95512-5fd7-4f30-9ef0-b495ed6a437c" containerName="revoke-keys" Oct 30 00:46:35.445487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.445470 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a95512-5fd7-4f30-9ef0-b495ed6a437c" containerName="revoke-keys" Oct 30 00:46:35.458399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.458368 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:46:35.458834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.458814 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574589 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574643 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574743 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574804 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574838 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574918 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.574962 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.575071 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.575362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.575133 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrxz\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.677123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.677028 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.677396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.677370 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.677544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.677526 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.677696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.677678 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.677874 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.677837 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.678095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.678070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.679775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.679751 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrxz\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.679936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.679910 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.680023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.679632 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.680023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.678588 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.680140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.679219 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.680611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.680586 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.681099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.681078 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.681559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.681024 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.687910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.686311 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.688450 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.688073 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.693662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.693636 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:35.705712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.705056 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.705712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.705650 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrxz\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.791885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.791465 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:35.843298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:35.843206 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:36.205805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:36.205761 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:36.207734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:36.207705 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:36.207936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:36.207912 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:46:36.236484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:36.236457 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:46:37.212029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.211989 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" event={"ID":"3ae85fb7-5635-4154-a571-6f1c40a533d2","Type":"ContainerStarted","Data":"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154"} Oct 30 00:46:37.212576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.212552 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" event={"ID":"3ae85fb7-5635-4154-a571-6f1c40a533d2","Type":"ContainerStarted","Data":"a677d4e04842e7ea6572a1eb768aca14b6e3eb69c728c33eac6cea84dc5df5d4"} Oct 30 00:46:37.312659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.312586 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podStartSLOduration=2.312553922 podStartE2EDuration="2.312553922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:37.27202281 +0000 UTC m=+849.512257924" watchObservedRunningTime="2026-10-30 00:46:37.312553922 +0000 UTC m=+849.552789035" Oct 30 00:46:37.602901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.602858 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:37.794318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.794285 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:37.802173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.802141 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:37.802422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:37.802393 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:38.801888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:38.796153 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:38.801888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:38.796217 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:38.922753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:38.922720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:39.371002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:39.370964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:39.800942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:39.797132 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:39.800942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:39.797190 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:40.799210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:40.798786 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:40.799210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:40.798927 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:41.043388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:41.043357 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:41.796198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:41.796160 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:41.796490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:41.796461 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:42.677785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:42.677747 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:42.792652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:42.792612 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 00:46:42.792820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:42.792694 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 00:46:44.147142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:44.147112 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:44.147655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:44.147628 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:44.148478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:44.148460 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:46:44.470204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:44.470118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:45.999317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:45.999283 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:46.555288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.555257 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5"] Oct 30 00:46:46.559305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.559281 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.574456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.574433 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:46:46.605883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.597777 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:46:46.606172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.606154 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:46:46.609073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.609046 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:46:46.610392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.610368 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5"] Oct 30 00:46:46.675588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.675566 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.675724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.675606 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.675791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.675748 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8shs\" (UniqueName: \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.776798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.776767 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8shs\" (UniqueName: \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.776949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.776893 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.777022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.776946 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.777640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.777623 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.779211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.779193 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.815071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.815021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8shs\" (UniqueName: \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\") pod \"maas-api-revoke-sub-cf3c2dcc-24cd5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:46.874055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:46.874032 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:47.177207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:47.177177 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5"] Oct 30 00:46:47.178225 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:46:47.178193 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b04b3ff_1670_47b5_aff4_3913b1a5f5e5.slice/crio-733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2 WatchSource:0}: Error finding container 733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2: Status 404 returned error can't find the container with id 733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2 Oct 30 00:46:47.301072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:47.301044 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" event={"ID":"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5","Type":"ContainerStarted","Data":"733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2"} Oct 30 00:46:47.737225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:47.737188 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:48.309840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:48.309807 5938 generic.go:356] "Generic (PLEG): container finished" podID="5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" containerID="8ccbe316a49b0b6d72a1bbe826bc434ab5b2a9015f6cad196bb31de4821d4693" exitCode=0 Oct 30 00:46:48.310256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:48.309887 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" event={"ID":"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5","Type":"ContainerDied","Data":"8ccbe316a49b0b6d72a1bbe826bc434ab5b2a9015f6cad196bb31de4821d4693"} Oct 30 00:46:49.364331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.364288 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:49.509383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.509356 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:49.610643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.610614 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\") pod \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " Oct 30 00:46:49.610797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.610785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\") pod \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " Oct 30 00:46:49.610858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.610818 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\" (UniqueName: \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\") pod \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\" (UID: \"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5\") " Oct 30 00:46:49.611009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.610988 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca" pod "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" (UID: "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:46:49.611292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.611272 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:49.613169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.613138 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs" pod "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" (UID: "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5"). InnerVolumeSpecName "kube-api-access-k8shs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:46:49.613256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.613208 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca" pod "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" (UID: "5b04b3ff-1670-47b5-aff4-3913b1a5f5e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:46:49.712451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.712427 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:49.712451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:49.712449 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8shs\" (UniqueName: \"kubernetes.io/projected/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5-kube-api-access-k8shs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:46:50.324400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:50.324320 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" Oct 30 00:46:50.324400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:50.324355 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5" event={"ID":"5b04b3ff-1670-47b5-aff4-3913b1a5f5e5","Type":"ContainerDied","Data":"733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2"} Oct 30 00:46:50.324571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:50.324412 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733bf61b23c9792559b851b7d558766d992138a81e2a168757f2aef84fdee0b2" Oct 30 00:46:50.874801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:50.874768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:52.491078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:52.491042 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:54.067459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:54.067418 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:56.077987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:56.077945 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:56.515409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:56.515374 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:58.150295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:58.150266 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:58.837806 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:46:58.837769 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:46:59.882880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:46:59.882845 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:01.492549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:01.489969 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:02.981276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:02.981242 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:03.331282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.331199 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h"] Oct 30 00:47:03.332446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.332418 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" containerName="revoke-keys" Oct 30 00:47:03.332446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.332444 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" containerName="revoke-keys" Oct 30 00:47:03.332807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.332791 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" containerName="revoke-keys" Oct 30 00:47:03.338744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.338722 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.350642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.350617 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h"] Oct 30 00:47:03.359100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.359074 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:03.359241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.359137 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:03.364333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.364312 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:03.364441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.364412 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:03.407465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.407441 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.407581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.407495 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gqs\" (UniqueName: \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.407728 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.407709 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.508780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.508756 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.508940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.508901 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.509023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.508971 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gqs\" (UniqueName: \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.509555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.509536 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.510815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.510798 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.531178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.531158 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gqs\" (UniqueName: \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\") pod \"maas-api-revoke-sub-cfba6cfd-lb94h\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.655583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.655563 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:03.959048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:03.959021 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h"] Oct 30 00:47:03.960225 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:03.960195 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b42a07_b334_4aee_a379_aaaccce18c94.slice/crio-4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084 WatchSource:0}: Error finding container 4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084: Status 404 returned error can't find the container with id 4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084 Oct 30 00:47:04.438366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:04.438328 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" event={"ID":"b5b42a07-b334-4aee-a379-aaaccce18c94","Type":"ContainerStarted","Data":"f63d9236231717ec73b708a84e2cb045e4eb5f1311ea954e9b4076db57fa1bc9"} Oct 30 00:47:04.438699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:04.438391 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" event={"ID":"b5b42a07-b334-4aee-a379-aaaccce18c94","Type":"ContainerStarted","Data":"4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084"} Oct 30 00:47:04.478760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:04.478720 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" podStartSLOduration=1.478705935 podStartE2EDuration="1.478705935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:04.476281237 +0000 UTC m=+876.716516350" watchObservedRunningTime="2026-10-30 00:47:04.478705935 +0000 UTC m=+876.718941047" Oct 30 00:47:04.591172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:04.591109 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:05.446178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:05.446144 5938 generic.go:356] "Generic (PLEG): container finished" podID="b5b42a07-b334-4aee-a379-aaaccce18c94" containerID="f63d9236231717ec73b708a84e2cb045e4eb5f1311ea954e9b4076db57fa1bc9" exitCode=0 Oct 30 00:47:05.446568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:05.446227 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" event={"ID":"b5b42a07-b334-4aee-a379-aaaccce18c94","Type":"ContainerDied","Data":"f63d9236231717ec73b708a84e2cb045e4eb5f1311ea954e9b4076db57fa1bc9"} Oct 30 00:47:06.181625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.181590 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:06.612732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.612702 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:06.745977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.745889 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\") pod \"b5b42a07-b334-4aee-a379-aaaccce18c94\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " Oct 30 00:47:06.746153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.745981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\" (UniqueName: \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\") pod \"b5b42a07-b334-4aee-a379-aaaccce18c94\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " Oct 30 00:47:06.746275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.746255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\") pod \"b5b42a07-b334-4aee-a379-aaaccce18c94\" (UID: \"b5b42a07-b334-4aee-a379-aaaccce18c94\") " Oct 30 00:47:06.746684 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.746656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca" pod "b5b42a07-b334-4aee-a379-aaaccce18c94" (UID: "b5b42a07-b334-4aee-a379-aaaccce18c94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:06.748115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.748088 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca" pod "b5b42a07-b334-4aee-a379-aaaccce18c94" (UID: "b5b42a07-b334-4aee-a379-aaaccce18c94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:06.748312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.748290 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs" pod "b5b42a07-b334-4aee-a379-aaaccce18c94" (UID: "b5b42a07-b334-4aee-a379-aaaccce18c94"). InnerVolumeSpecName "kube-api-access-v2gqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:06.847668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.847629 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:06.847668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.847665 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2gqs\" (UniqueName: \"kubernetes.io/projected/b5b42a07-b334-4aee-a379-aaaccce18c94-kube-api-access-v2gqs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:06.847668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:06.847675 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b42a07-b334-4aee-a379-aaaccce18c94-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:07.472548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:07.471233 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" Oct 30 00:47:07.472548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:07.472481 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h" event={"ID":"b5b42a07-b334-4aee-a379-aaaccce18c94","Type":"ContainerDied","Data":"4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084"} Oct 30 00:47:07.472814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:07.472581 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de4f9e307272d25405a89b0d9989ae87a590884f53beddd0fb071d7829b4084" Oct 30 00:47:08.061440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:08.061407 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:09.678012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:09.677944 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:09.738028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:09.737978 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:09.804004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:09.803961 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:09.904053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:09.904014 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:10.348434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:10.348396 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:11.858958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:11.858922 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:13.343507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:13.343473 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.978145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:14.978105 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:15.667508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.667473 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth"] Oct 30 00:47:15.668118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.668102 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b5b42a07-b334-4aee-a379-aaaccce18c94" containerName="revoke-keys" Oct 30 00:47:15.668118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.668118 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b42a07-b334-4aee-a379-aaaccce18c94" containerName="revoke-keys" Oct 30 00:47:15.668263 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.668211 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b42a07-b334-4aee-a379-aaaccce18c94" containerName="revoke-keys" Oct 30 00:47:15.671637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.671621 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.685621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.685597 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:15.689541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.689520 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth"] Oct 30 00:47:15.697250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.697231 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:15.697381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.697360 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:15.697459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.697297 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:15.743854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.743826 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.743988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.743969 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r69d\" (UniqueName: \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.744034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.744002 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.845038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.845013 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.845137 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.845083 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r69d\" (UniqueName: \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.845137 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.845104 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.845722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.845705 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.847292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.847273 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.867627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.867604 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r69d\" (UniqueName: \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\") pod \"maas-api-revoke-sub-11b081af-n5hth\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:15.985607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:15.985554 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:16.288220 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:16.288195 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a21b2a_d62f_42c4_b64d_4d1fe2e7a682.slice/crio-25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8 WatchSource:0}: Error finding container 25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8: Status 404 returned error can't find the container with id 25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8 Oct 30 00:47:16.288370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:16.288346 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth"] Oct 30 00:47:16.554584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:16.554526 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" event={"ID":"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682","Type":"ContainerStarted","Data":"25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8"} Oct 30 00:47:16.677732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:16.677700 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:17.568313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:17.568229 5938 generic.go:356] "Generic (PLEG): container finished" podID="03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" containerID="3fdfb30fd2d40e345b740e7b5e0f39836eef9df80fce3d07ee03d41e79266998" exitCode=0 Oct 30 00:47:17.568673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:17.568318 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" event={"ID":"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682","Type":"ContainerDied","Data":"3fdfb30fd2d40e345b740e7b5e0f39836eef9df80fce3d07ee03d41e79266998"} Oct 30 00:47:18.188081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.188050 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:18.722819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.722795 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:18.783003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.782974 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\") pod \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " Oct 30 00:47:18.783176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.783053 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\" (UniqueName: \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\") pod \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " Oct 30 00:47:18.783176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.783109 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\") pod \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\" (UID: \"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682\") " Oct 30 00:47:18.783325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.783302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca" pod "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" (UID: "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:18.783732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.783705 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:18.785065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.785035 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca" pod "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" (UID: "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:18.785185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.785149 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d" pod "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" (UID: "03a21b2a-d62f-42c4-b64d-4d1fe2e7a682"). InnerVolumeSpecName "kube-api-access-9r69d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:18.851524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.851467 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79"] Oct 30 00:47:18.852105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.852090 5938 cpu_manager.go:414] "containerMap: removing container" podUID="03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" containerName="revoke-keys" Oct 30 00:47:18.852172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.852106 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" containerName="revoke-keys" Oct 30 00:47:18.852209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.852197 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" containerName="revoke-keys" Oct 30 00:47:18.855677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.855661 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.875016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.874991 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79"] Oct 30 00:47:18.884526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.884500 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.884639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.884614 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.884813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.884785 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2g5\" (UniqueName: \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.885112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.885091 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r69d\" (UniqueName: \"kubernetes.io/projected/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-kube-api-access-9r69d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:18.885213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.885116 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:18.986653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.986621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.986798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.986680 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.986896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.986840 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2g5\" (UniqueName: \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.987431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.987411 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:18.989102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:18.989080 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:19.009109 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.009080 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2g5\" (UniqueName: \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\") pod \"maas-api-revoke-sub-bd1e492a-9mq79\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:19.169400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.169377 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:19.518590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.518566 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79"] Oct 30 00:47:19.520423 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:19.520399 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ce8980_713b_482c_8717_946007219012.slice/crio-e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c WatchSource:0}: Error finding container e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c: Status 404 returned error can't find the container with id e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c Oct 30 00:47:19.581519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.581497 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" event={"ID":"03a21b2a-d62f-42c4-b64d-4d1fe2e7a682","Type":"ContainerDied","Data":"25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8"} Oct 30 00:47:19.581631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.581530 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth" Oct 30 00:47:19.581631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.581543 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25dbed3e8826cda78e27ff18fcc9b016ae28e846e047f4acd3a5774daa19a0d8" Oct 30 00:47:19.583034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.583000 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" event={"ID":"50ce8980-713b-482c-8717-946007219012","Type":"ContainerStarted","Data":"e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c"} Oct 30 00:47:19.729751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:19.729722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:20.465285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.465251 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:47:20.469364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.469339 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.494498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.494476 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:47:20.499666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.499638 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:47:20.504460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.504439 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.508716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508620 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.508716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508669 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjhl\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.508914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.508914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508810 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.508914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508840 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.508914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508894 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.509138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.508978 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.509138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.509050 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.509138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.509119 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.524044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.524019 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:47:20.590522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.590451 5938 generic.go:356] "Generic (PLEG): container finished" podID="50ce8980-713b-482c-8717-946007219012" containerID="9dc52395e3d009695a1721ba9d2b593ddcad63343f48da1d0a1ce5f2868ab4d8" exitCode=0 Oct 30 00:47:20.590650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.590535 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" event={"ID":"50ce8980-713b-482c-8717-946007219012","Type":"ContainerDied","Data":"9dc52395e3d009695a1721ba9d2b593ddcad63343f48da1d0a1ce5f2868ab4d8"} Oct 30 00:47:20.610009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.609982 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610061 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610080 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610096 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610118 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610308 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610341 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610371 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610482 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610546 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzzt\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610587 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610607 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610699 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610713 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610748 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.610920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.610998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.610966 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.611054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.611015 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.611054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.611041 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjhl\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.611174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.611117 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.611304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.611284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.611371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.611352 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.612735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.612717 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.612938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.612919 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.632055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.632026 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.635645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.635524 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjhl\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.712081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712051 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712083 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712110 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712175 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzzt\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712208 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712263 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712478 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712483 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712502 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.712848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712791 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.713003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712894 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.713003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.712947 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.714628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.714607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.714725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.714699 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.722773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.722750 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:47:20.728293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.728273 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.740130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.740102 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzzt\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.742214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.742189 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.764677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.764654 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:47:20.783376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.783351 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:20.814344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814179 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814243 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6h9b\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814426 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814539 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814591 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814679 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814698 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814724 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.814835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.814791 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.816321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.816300 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:47:20.821570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.821551 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.828340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.828305 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:20.859820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.859780 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:47:20.868640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.868609 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:47:20.885656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.885630 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:20.887456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.887431 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:47:20.916507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.916644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916522 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.916644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916582 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6h9b\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.916767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916679 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.916767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916703 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.916767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916753 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.916940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916778 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.916940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916888 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.916940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.917113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.916992 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.917169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917124 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.917169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917148 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917192 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917237 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917298 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917370 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917486 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.917529 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7577\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:20.918215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.918108 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.919578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.919189 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.919849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.919812 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.920348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.920323 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.920483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.920384 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.922021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.921286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.922021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.922576 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.943281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.942247 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:20.943281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:20.942353 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6h9b\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:21.018492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.018648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.018648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018599 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt66x\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.018648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018631 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.018775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7577\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.018775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.018775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018770 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.018945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018793 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.018945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.018945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018818 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.019099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.018993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.019099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019054 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019171 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019238 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.019328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019312 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019367 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.019386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019379 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019409 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019593 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019627 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.019715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019692 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.019984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.019964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.020914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.020895 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.021743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.021726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.041285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.041267 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.041496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.041471 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7577\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.079016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.078997 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:21.112821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.112798 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:47:21.113127 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:21.113095 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dba1ce4_fc03_4f62_bc06_ead6a364e3c3.slice/crio-60e81eaa73c28ec66e2d089476a38bbaeb5c7686efe09475a8ba4b924c702a8b WatchSource:0}: Error finding container 60e81eaa73c28ec66e2d089476a38bbaeb5c7686efe09475a8ba4b924c702a8b: Status 404 returned error can't find the container with id 60e81eaa73c28ec66e2d089476a38bbaeb5c7686efe09475a8ba4b924c702a8b Oct 30 00:47:21.115621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.115592 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.115687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.115660 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.115734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.115691 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.121391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121368 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121418 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121509 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121561 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121611 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121673 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.121791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121756 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt66x\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121793 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.121834 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.122136 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.122191 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.122353 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.122699 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.123525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.126848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.124336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.143394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.140730 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:21.147269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.147226 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.152275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.152224 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt66x\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.184219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.184200 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:47:21.193149 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:21.193114 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289013c3_a8f4_4dac_bcf9_1d7b293f2e2c.slice/crio-0fdcffb140c60347894e3d368fe192d14b98112a98f66df61f4425b67b7c5911 WatchSource:0}: Error finding container 0fdcffb140c60347894e3d368fe192d14b98112a98f66df61f4425b67b7c5911: Status 404 returned error can't find the container with id 0fdcffb140c60347894e3d368fe192d14b98112a98f66df61f4425b67b7c5911 Oct 30 00:47:21.196601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.196562 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.196743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.196635 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.196743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.196680 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.206422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.206238 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:21.399950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.399365 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:47:21.406943 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:21.406914 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode206ffd7_069f_4e1a_a4d0_31c5659f436c.slice/crio-9b6b555ee7e2f41648a1af772f4fffab43794f1a26e4dc428c86c47ff47ca02e WatchSource:0}: Error finding container 9b6b555ee7e2f41648a1af772f4fffab43794f1a26e4dc428c86c47ff47ca02e: Status 404 returned error can't find the container with id 9b6b555ee7e2f41648a1af772f4fffab43794f1a26e4dc428c86c47ff47ca02e Oct 30 00:47:21.412580 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.412506 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.412884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.412823 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.413163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.413109 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.477666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.477633 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:21.520054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.520024 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:47:21.541284 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:21.541263 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0b50c8_8bad_4637_89f0_a899b514d204.slice/crio-9c1ebfb36b727915c93f4b13557c2691b42c67a3912f3e044794823b75e29b5d WatchSource:0}: Error finding container 9c1ebfb36b727915c93f4b13557c2691b42c67a3912f3e044794823b75e29b5d: Status 404 returned error can't find the container with id 9c1ebfb36b727915c93f4b13557c2691b42c67a3912f3e044794823b75e29b5d Oct 30 00:47:21.543426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.543386 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.543540 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.543495 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.543603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.543547 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.616472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.615688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" event={"ID":"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3","Type":"ContainerStarted","Data":"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5"} Oct 30 00:47:21.616472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.615733 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" event={"ID":"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3","Type":"ContainerStarted","Data":"60e81eaa73c28ec66e2d089476a38bbaeb5c7686efe09475a8ba4b924c702a8b"} Oct 30 00:47:21.620131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.619327 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" event={"ID":"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c","Type":"ContainerStarted","Data":"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b"} Oct 30 00:47:21.620131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.619375 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" event={"ID":"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c","Type":"ContainerStarted","Data":"0fdcffb140c60347894e3d368fe192d14b98112a98f66df61f4425b67b7c5911"} Oct 30 00:47:21.622125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.622072 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" event={"ID":"6a0b50c8-8bad-4637-89f0-a899b514d204","Type":"ContainerStarted","Data":"9c1ebfb36b727915c93f4b13557c2691b42c67a3912f3e044794823b75e29b5d"} Oct 30 00:47:21.626911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.625341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" event={"ID":"e206ffd7-069f-4e1a-a4d0-31c5659f436c","Type":"ContainerStarted","Data":"3c5f41319580f0e755d8dd2e86cfc6c5d4cc6fda2ea9b37d1a9aa5dc0aa0626f"} Oct 30 00:47:21.626911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.625385 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" event={"ID":"e206ffd7-069f-4e1a-a4d0-31c5659f436c","Type":"ContainerStarted","Data":"9b6b555ee7e2f41648a1af772f4fffab43794f1a26e4dc428c86c47ff47ca02e"} Oct 30 00:47:21.626911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.625728 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:47:21.654896 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:21.654853 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944cdc20_56d4_41bf_9b47_4ee5e82b5d50.slice/crio-a5a3b29391a11c13b36c5a3c865573eb8b3e3628403bc82c8be571b72f0b1e47 WatchSource:0}: Error finding container a5a3b29391a11c13b36c5a3c865573eb8b3e3628403bc82c8be571b72f0b1e47: Status 404 returned error can't find the container with id a5a3b29391a11c13b36c5a3c865573eb8b3e3628403bc82c8be571b72f0b1e47 Oct 30 00:47:21.657116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.656893 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.657278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.657260 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.657387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.657371 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:21.766055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.763846 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podStartSLOduration=1.763826498 podStartE2EDuration="1.763826498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.694852059 +0000 UTC m=+893.935087173" watchObservedRunningTime="2026-10-30 00:47:21.763826498 +0000 UTC m=+894.004061616" Oct 30 00:47:21.772186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.770732 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podStartSLOduration=1.770722082 podStartE2EDuration="1.770722082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.762559804 +0000 UTC m=+894.002794918" watchObservedRunningTime="2026-10-30 00:47:21.770722082 +0000 UTC m=+894.010957196" Oct 30 00:47:21.783598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.783578 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:21.823451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.823064 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podStartSLOduration=1.823044705 podStartE2EDuration="1.823044705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.808716218 +0000 UTC m=+894.048951331" watchObservedRunningTime="2026-10-30 00:47:21.823044705 +0000 UTC m=+894.063279846" Oct 30 00:47:21.829984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.829956 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:21.962891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:21.959032 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:22.075538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.075505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\") pod \"50ce8980-713b-482c-8717-946007219012\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " Oct 30 00:47:22.075715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.075632 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\") pod \"50ce8980-713b-482c-8717-946007219012\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " Oct 30 00:47:22.075797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.075748 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\" (UniqueName: \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\") pod \"50ce8980-713b-482c-8717-946007219012\" (UID: \"50ce8980-713b-482c-8717-946007219012\") " Oct 30 00:47:22.076104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.076076 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca" pod "50ce8980-713b-482c-8717-946007219012" (UID: "50ce8980-713b-482c-8717-946007219012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:22.076281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.076263 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50ce8980-713b-482c-8717-946007219012-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:22.077954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.077915 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5" pod "50ce8980-713b-482c-8717-946007219012" (UID: "50ce8980-713b-482c-8717-946007219012"). InnerVolumeSpecName "kube-api-access-mt2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:22.079010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.078987 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca" pod "50ce8980-713b-482c-8717-946007219012" (UID: "50ce8980-713b-482c-8717-946007219012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:22.079148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.079127 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:22.080501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.080478 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:22.080603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.080528 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:22.177244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.177156 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2g5\" (UniqueName: \"kubernetes.io/projected/50ce8980-713b-482c-8717-946007219012-kube-api-access-mt2g5\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:22.177244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.177183 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50ce8980-713b-482c-8717-946007219012-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:22.675606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.675004 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" event={"ID":"6a0b50c8-8bad-4637-89f0-a899b514d204","Type":"ContainerStarted","Data":"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3"} Oct 30 00:47:22.680889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.678379 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" event={"ID":"944cdc20-56d4-41bf-9b47-4ee5e82b5d50","Type":"ContainerStarted","Data":"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11"} Oct 30 00:47:22.680889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.678436 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" event={"ID":"944cdc20-56d4-41bf-9b47-4ee5e82b5d50","Type":"ContainerStarted","Data":"a5a3b29391a11c13b36c5a3c865573eb8b3e3628403bc82c8be571b72f0b1e47"} Oct 30 00:47:22.743887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.739668 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" Oct 30 00:47:22.757886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.754826 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79" event={"ID":"50ce8980-713b-482c-8717-946007219012","Type":"ContainerDied","Data":"e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c"} Oct 30 00:47:22.757886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.754921 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e440838080586683b7837a785562a559ce47ea80a6915a928fdfba335345414c" Oct 30 00:47:22.774888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.770450 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podStartSLOduration=2.770431435 podStartE2EDuration="2.770431435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:22.73594484 +0000 UTC m=+894.976179955" watchObservedRunningTime="2026-10-30 00:47:22.770431435 +0000 UTC m=+895.010666549" Oct 30 00:47:22.837249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.837212 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:22.837495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.837470 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:22.851403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.851364 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:22.851593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.851430 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:22.876888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:22.873624 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podStartSLOduration=2.873604974 podStartE2EDuration="2.873604974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:22.851260537 +0000 UTC m=+895.091495651" watchObservedRunningTime="2026-10-30 00:47:22.873604974 +0000 UTC m=+895.113840086" Oct 30 00:47:23.135889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.131826 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.136214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.136179 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:23.141951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.141923 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:23.146019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.145987 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.146236 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.146210 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:23.211887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.211004 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:23.211887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.211136 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.211887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.211180 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:23.385889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.384424 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:23.613892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.609992 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:23.673889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.668977 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:23.746893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.741526 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:23.771032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.770986 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:23.803912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.784260 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.803912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.784316 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:23.820025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.819979 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:23.829295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.829258 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:23.829550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.829522 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:23.900197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.900099 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:23.997900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:23.996943 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.005895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.004946 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.116026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.115984 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:24.116280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.116248 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:24.143242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.143207 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:24.143474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.143446 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:24.166988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.166876 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:24.219180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.219145 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:24.219427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.219402 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:24.278466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.278420 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.493080 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.492956 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.515103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.515061 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.546237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.546193 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.610984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.610941 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.611899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.611856 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:47:24.673040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.672995 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.673630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.673608 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:47:24.931105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.931063 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:24.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.937303 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:24.972539 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.972510 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:47:24.981894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.972752 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:47:25.107889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.983838 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.171888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.146936 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:25.172140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:24.940620 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:25.172569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.009026 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.172739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.172714 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:25.172849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.171744 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.172976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.172954 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:25.176998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.176974 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.181767 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.185964 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.262201 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.479296 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.479377 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.341965 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.273464 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.480317 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:25.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.480612 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:47:25.507887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.503594 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:25.513040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.513009 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:47:25.530027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.521942 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.555764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.555734 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:47:25.645974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.621244 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.651316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.651273 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" podUID="671141fa-f7fe-4f3f-952e-bf6aebb07516" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.674984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.674940 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.706985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.706938 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:25.808957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.807145 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.808957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.807213 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:25.833887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.832962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:25.869448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.869393 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:25.869752 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:25.869719 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:26.030891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.027349 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:26.125888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.125498 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:26.125888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.125576 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:26.125888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.125666 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:47:26.176890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.166577 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:26.255889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.237210 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:26.277732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.277697 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:26.280969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.280931 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:26.285211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.281327 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:26.285385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.285359 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:26.520942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.518263 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:26.520942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.519672 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:47:26.521951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.521904 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:26.583349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.522093 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:26.583647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.562601 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:26.687027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.686968 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:26.705886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.699763 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:26.887306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.887267 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:26.887556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.887526 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:26.949656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.949617 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:26.950025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.949994 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:26.974423 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:47:26.974329 5938 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s" Oct 30 00:47:26.974741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:26.974724 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:47:27.116890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.096782 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:27.140944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.134145 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:27.161440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.134062 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.161978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.160991 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:27.188005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.187955 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:27.212979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.189405 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:27.240006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.239953 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.240979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.240950 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:27.260890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.248926 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:27.577948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.543461 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:47:27.593898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.547591 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.597891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.594253 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.621315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.600337 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.700103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.700056 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.700373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.642954 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.700991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.700954 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:27.984411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.984377 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:27.984683 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:27.984650 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:28.173894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.173000 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.173894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.173071 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:28.232891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.209949 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:47:28.232891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.210547 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.232891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.210588 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:28.232891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.213668 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.248956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.244088 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.263964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.263918 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.276891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.263940 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:28.277104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.266792 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:28.314087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.273617 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.314375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.314344 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:28.430904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.407972 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.592564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.592461 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.596738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.596655 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.638900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.626604 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.675952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.674487 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wpnlg" podUID="8a59a208-000a-448d-9ca1-6042908e69bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.700512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.700471 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.701086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.701052 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.712893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.702301 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.713316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.704930 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:28.798415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.798267 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.798415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.798325 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:28.931885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.923753 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:28.988858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:28.988791 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:29.098894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.095136 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:29.104919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.102769 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:29.192401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.192290 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:29.192974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.192931 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:29.256993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.238037 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.334883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.314933 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.334883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.316892 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:29.334883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.316953 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:29.334883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.317111 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.493991 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:47:29.485139 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:47:29.630143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.630090 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.631273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.631239 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.633213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.632074 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:29.711606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.701951 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.711606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.710943 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:47:29.751939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.742543 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:29.753565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.753508 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:29.796841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.796806 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:29.797129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.797097 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:29.833884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.833833 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:29.859146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:29.859106 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:30.116117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.116088 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:30.118325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.118288 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:30.142485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.142359 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:30.142485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.142415 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:30.244948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.216987 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:30.244948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.217052 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:30.248658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.248630 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:47:30.248933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.248901 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.260199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.260175 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:47:30.335893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.333568 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.335893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.333931 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:30.637142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.637099 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:30.637747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.637722 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:47:30.642645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.637945 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.702165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.702119 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.714795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.713958 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.786891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.763503 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:30.786891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.763927 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:30.786891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.786783 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:30.807924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.786845 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:30.843887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.835043 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:30.843887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:30.835104 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:31.100507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.100410 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:31.102350 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.102318 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:31.102565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.100738 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.188984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.187527 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:31.188984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.187590 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:31.213700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.213662 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:31.213912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.213721 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:31.273911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.268600 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.336017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.335952 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.336916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.336279 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.502438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.502399 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz"] Oct 30 00:47:31.513907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.503619 5938 cpu_manager.go:414] "containerMap: removing container" podUID="50ce8980-713b-482c-8717-946007219012" containerName="revoke-keys" Oct 30 00:47:31.661887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.656528 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.700622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.700578 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:31.773104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.733527 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:31.773104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.734780 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.773104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.739935 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.773104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.767289 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:31.773104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.767659 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:31.803343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.802362 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:31.803343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.802430 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:31.827193 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.530299 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ce8980-713b-482c-8717-946007219012" containerName="revoke-keys" Oct 30 00:47:31.827193 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.826381 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ce8980-713b-482c-8717-946007219012" containerName="revoke-keys" Oct 30 00:47:31.836926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.836611 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:31.839015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.838984 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:31.842978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.842941 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:31.863471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.863439 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:31.864803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.863922 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:31.865004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.864531 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:31.973890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.970237 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:31.982041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:31.978031 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vjsb\" (UniqueName: \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.358520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.158517 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:32.383947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.359099 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:32.384250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.356972 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:32.384807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.357019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.385313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.357522 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:32.385459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.328557 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.385915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.159435 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:32.410889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.386059 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:32.411160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.332157 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:32.411339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.411312 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:32.411460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.391201 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.411953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.411912 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz"] Oct 30 00:47:32.549886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.541448 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:32.549886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.544769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vjsb\" (UniqueName: \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.549886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.544821 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.619948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.573194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.739899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.621525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.739899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.659698 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.739899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.703633 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.739899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.714232 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.755046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.755003 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:32.785500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.756919 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:47:32.786486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.786455 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vjsb\" (UniqueName: \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\") pod \"maas-api-revoke-sub-46fea2a2-tqjnz\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:32.820911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.788551 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.856508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.789051 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.865929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.789088 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:32.887042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.789118 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:32.887042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.823099 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 00:47:32.887042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.885255 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 00:47:32.935680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.935633 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:32.935949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:32.935923 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:33.082462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.082344 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:33.082462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.082402 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:33.195943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.194958 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.272889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.263848 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:33.298890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.273802 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:33.305023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.304980 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:33.327494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.280019 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:33.327494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.308952 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:33.416263 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.392947 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.444231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.417813 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:33.444231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.418172 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.743887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.732892 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:33.743887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.734476 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.743887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.735199 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.798941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.755879 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:33.805988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.799037 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:33.866889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:33.859610 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:34.009889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.008187 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:34.025895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.017958 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 00:47:34.025895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.020059 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 00:47:34.025895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.020110 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 00:47:34.031071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.031041 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:47:34.145178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.145137 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:34.145400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.145366 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:34.253292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.253256 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 00:47:34.253532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.253502 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 00:47:34.258473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.255438 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:34.260240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.259688 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:34.404031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.400246 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:34.454968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.454911 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:34.456002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.455954 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:34.557038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.556997 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:34.668941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.664256 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:34.737920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.736277 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:34.786704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.786680 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:47:34.812949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.812920 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz"] Oct 30 00:47:34.813159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.813147 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:47:34.813270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.813261 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:47:34.823080 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.804031 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:34.846700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.846048 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:47:34.853375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.852941 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:34.853375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.852986 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:34.863313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:34.859981 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:35.042995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.042914 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:35.044852 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.044590 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:35.046461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.046338 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:35.056165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.056146 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:47:35.086154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.085529 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:47:35.087770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.087703 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 00:47:35.088118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.088092 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 00:47:35.168074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.167462 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 00:47:35.168074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.167523 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 00:47:35.433890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.375721 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:47:35.433890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.410263 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:35.528426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.506447 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:35.528426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.507541 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:35.528426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.512401 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:47:35.719041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.703932 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:47:35.754242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.737594 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:35.792226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.792100 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" event={"ID":"f0f0de89-a598-4902-9b83-89a274e87d44","Type":"ContainerStarted","Data":"e02756a7dda3b34ca0bb759acc60b5cf79bcf4d14997f880d1e87eeae0ca7c28"} Oct 30 00:47:35.792226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.792189 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" event={"ID":"f0f0de89-a598-4902-9b83-89a274e87d44","Type":"ContainerStarted","Data":"2df3e7638a420cfcd52a44b29d6dc869d2d74740748f383edfd697c4e095dbac"} Oct 30 00:47:35.834788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.834745 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:35.835818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.835041 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:35.874123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.873940 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:35.878854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.875145 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:35.878854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.875194 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:35.902906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.902875 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:35.914358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:35.914304 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:36.020936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.018148 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" podStartSLOduration=5.01812815 podStartE2EDuration="5.01812815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:35.917345336 +0000 UTC m=+908.157580450" watchObservedRunningTime="2026-10-30 00:47:36.01812815 +0000 UTC m=+908.258363264" Oct 30 00:47:36.061795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.061691 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:36.081188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.080294 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:36.121251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.109173 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:47:36.146902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.141776 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:36.357897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.356155 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:47:36.400886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.398034 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:47:36.443260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.442274 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:47:36.472599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.472570 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:36.816005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.811375 5938 generic.go:356] "Generic (PLEG): container finished" podID="f0f0de89-a598-4902-9b83-89a274e87d44" containerID="e02756a7dda3b34ca0bb759acc60b5cf79bcf4d14997f880d1e87eeae0ca7c28" exitCode=0 Oct 30 00:47:36.816005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.812764 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" event={"ID":"f0f0de89-a598-4902-9b83-89a274e87d44","Type":"ContainerDied","Data":"e02756a7dda3b34ca0bb759acc60b5cf79bcf4d14997f880d1e87eeae0ca7c28"} Oct 30 00:47:36.816692 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.816670 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:47:36.916061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.916029 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:36.916355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:36.916306 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:37.141565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.141523 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:37.141759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.141596 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:37.452451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.451964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:37.477414 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.477381 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:37.478700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.478678 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:47:37.983416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:37.983389 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:38.018259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.018232 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:38.019408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.019386 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:47:38.039341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.039310 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\" (UniqueName: \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\") pod \"f0f0de89-a598-4902-9b83-89a274e87d44\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " Oct 30 00:47:38.039642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.039532 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\") pod \"f0f0de89-a598-4902-9b83-89a274e87d44\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " Oct 30 00:47:38.039642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.039598 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\") pod \"f0f0de89-a598-4902-9b83-89a274e87d44\" (UID: \"f0f0de89-a598-4902-9b83-89a274e87d44\") " Oct 30 00:47:38.040742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.040487 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca" pod "f0f0de89-a598-4902-9b83-89a274e87d44" (UID: "f0f0de89-a598-4902-9b83-89a274e87d44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:38.042051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.042025 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca" pod "f0f0de89-a598-4902-9b83-89a274e87d44" (UID: "f0f0de89-a598-4902-9b83-89a274e87d44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:38.042253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.042231 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb" pod "f0f0de89-a598-4902-9b83-89a274e87d44" (UID: "f0f0de89-a598-4902-9b83-89a274e87d44"). InnerVolumeSpecName "kube-api-access-6vjsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:38.142355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.142322 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vjsb\" (UniqueName: \"kubernetes.io/projected/f0f0de89-a598-4902-9b83-89a274e87d44-kube-api-access-6vjsb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:38.142355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.142356 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:38.142584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.142369 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f0de89-a598-4902-9b83-89a274e87d44-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:38.829249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.829223 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" Oct 30 00:47:38.829415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.829253 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz" event={"ID":"f0f0de89-a598-4902-9b83-89a274e87d44","Type":"ContainerDied","Data":"2df3e7638a420cfcd52a44b29d6dc869d2d74740748f383edfd697c4e095dbac"} Oct 30 00:47:38.829415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:38.829300 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df3e7638a420cfcd52a44b29d6dc869d2d74740748f383edfd697c4e095dbac" Oct 30 00:47:39.054788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:39.054685 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:40.547421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:40.547389 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:41.422741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.422705 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l"] Oct 30 00:47:41.423306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.423290 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f0f0de89-a598-4902-9b83-89a274e87d44" containerName="revoke-keys" Oct 30 00:47:41.423306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.423306 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f0de89-a598-4902-9b83-89a274e87d44" containerName="revoke-keys" Oct 30 00:47:41.423441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.423404 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f0de89-a598-4902-9b83-89a274e87d44" containerName="revoke-keys" Oct 30 00:47:41.427316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.427296 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.443345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.443320 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l"] Oct 30 00:47:41.450239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.450192 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:41.450239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.450198 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:41.450401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.450285 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:41.453304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.453276 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:41.493351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.493323 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.493496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.493466 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nf4\" (UniqueName: \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.493567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.493551 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.594474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.594446 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.594920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.594546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nf4\" (UniqueName: \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.594920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.594608 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.595357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.595339 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.597169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.597146 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.616994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.616966 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nf4\" (UniqueName: \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\") pod \"maas-api-revoke-sub-a4afd8c5-lll9l\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:41.757720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:41.757388 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:42.276302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:42.276212 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l"] Oct 30 00:47:42.301976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:42.301935 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:43.069986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.057832 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" event={"ID":"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96","Type":"ContainerStarted","Data":"ad080283bb76cfde75c1b5c46db8b3b1fcd5c67f53900cd5744a1b55692667a2"} Oct 30 00:47:43.531960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.529528 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.592889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.589949 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.625768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.624532 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:47:43.646276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.646238 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.703893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.700394 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.703893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.701269 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.703893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.701367 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.733893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.725013 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:43.976013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.975975 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:43.976295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:43.976265 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.002118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.001966 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.002118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.002072 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.112888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.107946 5938 generic.go:356] "Generic (PLEG): container finished" podID="7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" containerID="9db9494a210cb2e7dfbaa20ae66209408996b1e6d8b3bf47aee39d62356c0aba" exitCode=0 Oct 30 00:47:44.112888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.108182 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" event={"ID":"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96","Type":"ContainerDied","Data":"9db9494a210cb2e7dfbaa20ae66209408996b1e6d8b3bf47aee39d62356c0aba"} Oct 30 00:47:44.186247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.186199 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.217203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.217164 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.272075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.270939 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:44.533067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.532952 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 00:47:44.533622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.533308 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.534023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.534002 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:47:44.598701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.598656 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:44.805258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.688726 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:44.812467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.703991 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.704038 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.705225 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.705671 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.805573 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.813533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.813574 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.813608 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:47:44.834886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.813640 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:47:44.870461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:44.870433 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:47:45.201100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.201042 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:47:45.214612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.210299 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:47:45.230773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.230734 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:45.237891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.234287 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:47:45.383256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.383215 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:47:45.404089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.404059 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:47:45.482086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.468780 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:45.565891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.557471 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:45.727997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.727519 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:45.819615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.819517 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:45.847066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.844053 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:47:45.854979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.848255 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:47:45.854979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.849513 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:47:45.881422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.872106 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:45.909926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.905657 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:47:45.914088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:45.912523 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:46.207154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.207111 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:46.302271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.302212 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:47:46.545467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.396235 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.478657 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.567575 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.567622 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.567673 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.567708 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:47:46.570161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.559153 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:47:46.577095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.576971 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:47:46.577095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.577019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:47:46.590954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.590316 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:47:46.612150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.612117 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:46.679282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.676398 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:47:46.679282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.677595 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:47:46.699404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.699381 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:47:46.703660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.703635 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\" (UniqueName: \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\") pod \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " Oct 30 00:47:46.703900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.703878 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\") pod \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " Oct 30 00:47:46.704186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.704166 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\") pod \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\" (UID: \"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96\") " Oct 30 00:47:46.707188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.705673 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:47:46.707649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.707618 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca" pod "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" (UID: "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:46.722366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.722341 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4" pod "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" (UID: "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96"). InnerVolumeSpecName "kube-api-access-j2nf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:46.736402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.735951 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca" pod "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" (UID: "7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:46.807110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.807034 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:46.807110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.807065 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nf4\" (UniqueName: \"kubernetes.io/projected/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-kube-api-access-j2nf4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:46.807110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:46.807079 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:47.093811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:47.093738 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:47.487780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:47.487749 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" Oct 30 00:47:47.488187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:47.487776 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l" event={"ID":"7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96","Type":"ContainerDied","Data":"ad080283bb76cfde75c1b5c46db8b3b1fcd5c67f53900cd5744a1b55692667a2"} Oct 30 00:47:47.488187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:47.487833 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad080283bb76cfde75c1b5c46db8b3b1fcd5c67f53900cd5744a1b55692667a2" Oct 30 00:47:48.893964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:48.893911 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:49.933312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.933280 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd"] Oct 30 00:47:49.933823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.933808 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" containerName="revoke-keys" Oct 30 00:47:49.933903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.933824 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" containerName="revoke-keys" Oct 30 00:47:49.933981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.933970 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" containerName="revoke-keys" Oct 30 00:47:49.937840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.937822 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:49.955255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.955235 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd"] Oct 30 00:47:49.962732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.962647 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:49.962732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.962679 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:49.962954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.962737 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:49.963505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:49.963484 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:50.045012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.044980 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.045118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.045018 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlq7\" (UniqueName: \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.045118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.045090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.146357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.146335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.146472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.146363 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlq7\" (UniqueName: \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.146472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.146387 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.147065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.147041 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.148603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.148585 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.169644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.169618 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlq7\" (UniqueName: \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\") pod \"maas-api-revoke-sub-ced525e4-dhkpd\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.252236 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.252185 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:50.459923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.459871 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:50.564919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:50.564892 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd"] Oct 30 00:47:50.565523 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:50.565497 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb96e3_cdc9_4ea3_a633_2cb44101a84a.slice/crio-63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c WatchSource:0}: Error finding container 63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c: Status 404 returned error can't find the container with id 63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c Oct 30 00:47:51.007812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.007781 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:47:51.012071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.012052 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.045256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.045164 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:47:51.057515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057481 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057550 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057608 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057659 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057690 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057725 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057771 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.057947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.057799 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.058260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.058236 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8njhk\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160056 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8njhk\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160155 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160283 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160318 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160359 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160552 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160682 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160662 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160686 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160760 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.160952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160933 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.161014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.160963 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.162554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.162535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.162803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.162786 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.182257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.182232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8njhk\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.182347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.182234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.326171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.326142 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:51.523715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.523688 5938 generic.go:356] "Generic (PLEG): container finished" podID="18bb96e3-cdc9-4ea3-a633-2cb44101a84a" containerID="5c41ad629738a9f58e0231d1ba408e40eba1f0bedd4dacc740cd41c177ac7af2" exitCode=0 Oct 30 00:47:51.523855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.523727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" event={"ID":"18bb96e3-cdc9-4ea3-a633-2cb44101a84a","Type":"ContainerDied","Data":"5c41ad629738a9f58e0231d1ba408e40eba1f0bedd4dacc740cd41c177ac7af2"} Oct 30 00:47:51.523855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.523770 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" event={"ID":"18bb96e3-cdc9-4ea3-a633-2cb44101a84a","Type":"ContainerStarted","Data":"63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c"} Oct 30 00:47:51.644646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.644613 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:47:51.644802 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:51.644782 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891703a9_fa46_4691_89ec_7a7777c31f45.slice/crio-1a7ac1fefacd1625652be8adbcda9b58ce8f581899c68c813145bc8fe59a657d WatchSource:0}: Error finding container 1a7ac1fefacd1625652be8adbcda9b58ce8f581899c68c813145bc8fe59a657d: Status 404 returned error can't find the container with id 1a7ac1fefacd1625652be8adbcda9b58ce8f581899c68c813145bc8fe59a657d Oct 30 00:47:51.647307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.647275 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:51.647394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.647340 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:51.647394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:51.647372 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:47:52.163574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.163536 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:52.533582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.533486 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" event={"ID":"891703a9-fa46-4691-89ec-7a7777c31f45","Type":"ContainerStarted","Data":"02594ba4bb29dce5d94014c888769ab56cc4fff359d51177fd75ad6eee661860"} Oct 30 00:47:52.533582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.533539 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" event={"ID":"891703a9-fa46-4691-89ec-7a7777c31f45","Type":"ContainerStarted","Data":"1a7ac1fefacd1625652be8adbcda9b58ce8f581899c68c813145bc8fe59a657d"} Oct 30 00:47:52.614884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.612108 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podStartSLOduration=2.612082934 podStartE2EDuration="2.612082934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:52.596425348 +0000 UTC m=+924.836660462" watchObservedRunningTime="2026-10-30 00:47:52.612082934 +0000 UTC m=+924.852318058" Oct 30 00:47:52.769946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.769920 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:52.918355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.918323 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\" (UniqueName: \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\") pod \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " Oct 30 00:47:52.918543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.918367 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\") pod \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " Oct 30 00:47:52.918591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.918553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\") pod \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\" (UID: \"18bb96e3-cdc9-4ea3-a633-2cb44101a84a\") " Oct 30 00:47:52.918833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.918809 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca" pod "18bb96e3-cdc9-4ea3-a633-2cb44101a84a" (UID: "18bb96e3-cdc9-4ea3-a633-2cb44101a84a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:47:52.919173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.919148 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:52.920581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.920557 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca" pod "18bb96e3-cdc9-4ea3-a633-2cb44101a84a" (UID: "18bb96e3-cdc9-4ea3-a633-2cb44101a84a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:47:52.920581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:52.920564 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7" pod "18bb96e3-cdc9-4ea3-a633-2cb44101a84a" (UID: "18bb96e3-cdc9-4ea3-a633-2cb44101a84a"). InnerVolumeSpecName "kube-api-access-7wlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:47:53.020903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.020849 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:53.020903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.020898 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wlq7\" (UniqueName: \"kubernetes.io/projected/18bb96e3-cdc9-4ea3-a633-2cb44101a84a-kube-api-access-7wlq7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:47:53.326573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.326492 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:47:53.328199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.328175 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:53.328316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.328221 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:53.541824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.541798 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" Oct 30 00:47:53.542033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.541798 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd" event={"ID":"18bb96e3-cdc9-4ea3-a633-2cb44101a84a","Type":"ContainerDied","Data":"63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c"} Oct 30 00:47:53.542033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.541939 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a0358740132e745d57c2fea82db9a2c33f20c153ebf999bdb4cbcc1559609c" Oct 30 00:47:53.882818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:53.882782 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:54.327004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:54.326912 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:54.327004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:54.326973 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:55.327100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:55.327070 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:55.327512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:55.327118 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:55.657112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:55.657079 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:56.326616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.326577 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:56.326809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.326643 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:56.874329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.874297 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp"] Oct 30 00:47:56.874909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.874895 5938 cpu_manager.go:414] "containerMap: removing container" podUID="18bb96e3-cdc9-4ea3-a633-2cb44101a84a" containerName="revoke-keys" Oct 30 00:47:56.874909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.874909 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bb96e3-cdc9-4ea3-a633-2cb44101a84a" containerName="revoke-keys" Oct 30 00:47:56.875016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.875006 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bb96e3-cdc9-4ea3-a633-2cb44101a84a" containerName="revoke-keys" Oct 30 00:47:56.879164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.879147 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:56.922910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.922880 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp"] Oct 30 00:47:56.927782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.927752 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:47:56.928057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.928035 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:47:56.928236 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.928172 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:47:56.928362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.927970 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:47:56.977062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.977035 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:56.977385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.977360 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbr7h\" (UniqueName: \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:56.977541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:56.977522 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.078570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.078536 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbr7h\" (UniqueName: \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.078728 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.078579 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.078728 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.078621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.079359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.079336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.080980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.080958 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.110925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.110680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbr7h\" (UniqueName: \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\") pod \"maas-api-revoke-sub-a719eeaa-54xsp\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.231385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.231310 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:47:57.329512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.327469 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:57.329512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.327538 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:57.474813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.474783 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:57.568782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:57.568736 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp"] Oct 30 00:47:57.572240 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:47:57.572191 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75c5579_9aba_4b05_89e1_c87f3195e417.slice/crio-67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c WatchSource:0}: Error finding container 67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c: Status 404 returned error can't find the container with id 67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c Oct 30 00:47:58.327382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:58.327352 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 00:47:58.327838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:58.327409 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 00:47:58.607893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:58.607845 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" event={"ID":"c75c5579-9aba-4b05-89e1-c87f3195e417","Type":"ContainerStarted","Data":"af44c5fb3d1e04d35dd31bca2c0e0582e7148071403a6c9adb7c067c2f459119"} Oct 30 00:47:58.608184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:58.608149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" event={"ID":"c75c5579-9aba-4b05-89e1-c87f3195e417","Type":"ContainerStarted","Data":"67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c"} Oct 30 00:47:58.651954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:58.651896 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" podStartSLOduration=2.651881964 podStartE2EDuration="2.651881964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:58.647195494 +0000 UTC m=+930.887430618" watchObservedRunningTime="2026-10-30 00:47:58.651881964 +0000 UTC m=+930.892117086" Oct 30 00:47:58.912985 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:47:58.912892 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:47:59.687444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:59.687395 5938 generic.go:356] "Generic (PLEG): container finished" podID="c75c5579-9aba-4b05-89e1-c87f3195e417" containerID="af44c5fb3d1e04d35dd31bca2c0e0582e7148071403a6c9adb7c067c2f459119" exitCode=0 Oct 30 00:47:59.688092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:47:59.688048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" event={"ID":"c75c5579-9aba-4b05-89e1-c87f3195e417","Type":"ContainerDied","Data":"af44c5fb3d1e04d35dd31bca2c0e0582e7148071403a6c9adb7c067c2f459119"} Oct 30 00:48:00.293888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.293670 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:00.351929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.351308 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.374890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.371671 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:00.374890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.371733 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.467896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.467007 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.586944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.582958 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.609888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.598939 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:48:00.697890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.696062 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.697890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.696418 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:00.840941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.839951 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:00.885916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.872972 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:48:00.922887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:00.921946 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:48:01.134282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.133947 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.468315 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.472004 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.474370 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.475791 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.503909 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.504895 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:01.510920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.504940 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:01.696901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.694336 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:01.729890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.720978 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:01.762947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.749094 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:48:01.794890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.765339 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:48:01.794890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.765637 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:01.794890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.768348 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:01.794890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.768407 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:01.794890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.768441 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.845510 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.847264 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.847309 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:01.848297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.012992 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:02.173101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.036098 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:48:02.455840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.435826 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:02.480996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.456181 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:02.565929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.475264 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:02.622934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.481426 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:02.624426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.481457 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.663563 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.773413 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.773658 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.793973 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.794885 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.795113 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:48:02.853045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.795768 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:02.885311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.872953 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:48:02.885311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:02.878892 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:03.074122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.073991 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:03.074317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.074296 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:03.081523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.081475 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:03.128200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.128163 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:03.170460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.168233 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:03.191703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.191617 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:03.192496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.192414 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:03.197470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.197450 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:03.220804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.220784 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:03.233314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.231442 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:03.233314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.232987 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:03.233314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.233033 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:03.235363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.234166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:03.251875 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.248909 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:48:03.261900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.261687 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:03.374580 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.374507 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" Oct 30 00:48:03.374853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.374564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp" event={"ID":"c75c5579-9aba-4b05-89e1-c87f3195e417","Type":"ContainerDied","Data":"67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c"} Oct 30 00:48:03.375044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.375028 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67abbbcd6e17b71503bad529c8a7144a961cac749c46d1e95c7f5e131976e96c" Oct 30 00:48:03.376437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.376408 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\" (UniqueName: \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\") pod \"c75c5579-9aba-4b05-89e1-c87f3195e417\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " Oct 30 00:48:03.377418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.377392 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\") pod \"c75c5579-9aba-4b05-89e1-c87f3195e417\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " Oct 30 00:48:03.377644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.377617 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\") pod \"c75c5579-9aba-4b05-89e1-c87f3195e417\" (UID: \"c75c5579-9aba-4b05-89e1-c87f3195e417\") " Oct 30 00:48:03.378975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.378949 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca" pod "c75c5579-9aba-4b05-89e1-c87f3195e417" (UID: "c75c5579-9aba-4b05-89e1-c87f3195e417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:03.379073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.379046 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h" pod "c75c5579-9aba-4b05-89e1-c87f3195e417" (UID: "c75c5579-9aba-4b05-89e1-c87f3195e417"). InnerVolumeSpecName "kube-api-access-kbr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:03.381630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.381575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca" pod "c75c5579-9aba-4b05-89e1-c87f3195e417" (UID: "c75c5579-9aba-4b05-89e1-c87f3195e417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:03.479594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.479551 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbr7h\" (UniqueName: \"kubernetes.io/projected/c75c5579-9aba-4b05-89e1-c87f3195e417-kube-api-access-kbr7h\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:03.479594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.479589 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:03.479832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.479605 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c5579-9aba-4b05-89e1-c87f3195e417-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:03.483068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.483041 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:48:03.483375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.483348 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:48:03.484129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:03.484112 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:48:04.043829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.043742 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd"] Oct 30 00:48:04.044587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.044564 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c75c5579-9aba-4b05-89e1-c87f3195e417" containerName="revoke-keys" Oct 30 00:48:04.044693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.044588 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75c5579-9aba-4b05-89e1-c87f3195e417" containerName="revoke-keys" Oct 30 00:48:04.044749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.044736 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75c5579-9aba-4b05-89e1-c87f3195e417" containerName="revoke-keys" Oct 30 00:48:04.048192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.048168 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.062439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.062343 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:04.062547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.062445 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:04.062664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.062560 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:04.062664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.062627 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:04.065892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.065842 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd"] Oct 30 00:48:04.087765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.087737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.087942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.087926 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tfx\" (UniqueName: \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.088043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.088028 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.189406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.189381 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tfx\" (UniqueName: \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.189499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.189460 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.189583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.189568 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.190212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.190188 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.191747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.191729 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.212066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.212043 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tfx\" (UniqueName: \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\") pod \"maas-api-revoke-sub-63f65bdf-xr8wd\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.362027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.362006 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:04.571496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.571456 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:04.681926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:04.681901 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd"] Oct 30 00:48:04.683306 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:04.683277 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7078218e_0204_439a_8d2e_19847cdcc867.slice/crio-3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337 WatchSource:0}: Error finding container 3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337: Status 404 returned error can't find the container with id 3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337 Oct 30 00:48:05.404851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:05.404822 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" event={"ID":"7078218e-0204-439a-8d2e-19847cdcc867","Type":"ContainerStarted","Data":"44a2cd248d81c82ba6da4c7e13a174280b4dbf3994ff1223ae709aa51e6ebe7f"} Oct 30 00:48:05.405234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:05.404902 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" event={"ID":"7078218e-0204-439a-8d2e-19847cdcc867","Type":"ContainerStarted","Data":"3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337"} Oct 30 00:48:05.444051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:05.444001 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" podStartSLOduration=1.443987902 podStartE2EDuration="1.443987902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:05.442564872 +0000 UTC m=+937.682799990" watchObservedRunningTime="2026-10-30 00:48:05.443987902 +0000 UTC m=+937.684223016" Oct 30 00:48:06.049123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:06.049083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:06.191947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:06.191913 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:06.413133 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:06.413102 5938 generic.go:356] "Generic (PLEG): container finished" podID="7078218e-0204-439a-8d2e-19847cdcc867" containerID="44a2cd248d81c82ba6da4c7e13a174280b4dbf3994ff1223ae709aa51e6ebe7f" exitCode=0 Oct 30 00:48:06.413503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:06.413192 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" event={"ID":"7078218e-0204-439a-8d2e-19847cdcc867","Type":"ContainerDied","Data":"44a2cd248d81c82ba6da4c7e13a174280b4dbf3994ff1223ae709aa51e6ebe7f"} Oct 30 00:48:07.538951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.538918 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:07.580530 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.580484 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:07.633253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.633217 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\" (UniqueName: \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\") pod \"7078218e-0204-439a-8d2e-19847cdcc867\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " Oct 30 00:48:07.633421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.633398 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\") pod \"7078218e-0204-439a-8d2e-19847cdcc867\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " Oct 30 00:48:07.633573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.633551 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\") pod \"7078218e-0204-439a-8d2e-19847cdcc867\" (UID: \"7078218e-0204-439a-8d2e-19847cdcc867\") " Oct 30 00:48:07.633921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.633892 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca" pod "7078218e-0204-439a-8d2e-19847cdcc867" (UID: "7078218e-0204-439a-8d2e-19847cdcc867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:07.634420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.634395 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:07.636316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.636284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca" pod "7078218e-0204-439a-8d2e-19847cdcc867" (UID: "7078218e-0204-439a-8d2e-19847cdcc867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:07.641980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.641959 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx" pod "7078218e-0204-439a-8d2e-19847cdcc867" (UID: "7078218e-0204-439a-8d2e-19847cdcc867"). InnerVolumeSpecName "kube-api-access-t9tfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:07.735760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.735709 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7078218e-0204-439a-8d2e-19847cdcc867-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:07.735760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:07.735731 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tfx\" (UniqueName: \"kubernetes.io/projected/7078218e-0204-439a-8d2e-19847cdcc867-kube-api-access-t9tfx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:08.429008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.428972 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" event={"ID":"7078218e-0204-439a-8d2e-19847cdcc867","Type":"ContainerDied","Data":"3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337"} Oct 30 00:48:08.429008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.429008 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd" Oct 30 00:48:08.429214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.429033 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b10af9fd9b6844594c5306e82d783fb0768e03ec91dabffda8debfc6278a337" Oct 30 00:48:08.527956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.527924 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql"] Oct 30 00:48:08.528494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.528480 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7078218e-0204-439a-8d2e-19847cdcc867" containerName="revoke-keys" Oct 30 00:48:08.528534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.528496 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7078218e-0204-439a-8d2e-19847cdcc867" containerName="revoke-keys" Oct 30 00:48:08.528620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.528612 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7078218e-0204-439a-8d2e-19847cdcc867" containerName="revoke-keys" Oct 30 00:48:08.532377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.532354 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.547942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.547921 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:08.548368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.547997 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:08.548368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.548026 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:08.553160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.553135 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql"] Oct 30 00:48:08.558791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.558769 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:08.647157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.647131 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8fw\" (UniqueName: \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.647278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.647184 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.647335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.647314 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.749088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.749027 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8fw\" (UniqueName: \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.749088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.749075 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.749234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.749131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.749700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.749671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.751470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.751451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.771071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.771047 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8fw\" (UniqueName: \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\") pod \"maas-api-revoke-sub-8afe1aed-sspql\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:08.846274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:08.846253 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:09.147344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:09.147313 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql"] Oct 30 00:48:09.147483 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:09.147427 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9122261f_ea39_4ed3_834c_a4e0c7a2aa0d.slice/crio-f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c WatchSource:0}: Error finding container f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c: Status 404 returned error can't find the container with id f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c Oct 30 00:48:09.282462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:09.282431 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:09.437088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:09.437062 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" event={"ID":"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d","Type":"ContainerStarted","Data":"0e671fd6632b0759cc7b4b86195d9b036dfd3d77a467912bbb91ffe61208bbe8"} Oct 30 00:48:09.437197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:09.437116 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" event={"ID":"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d","Type":"ContainerStarted","Data":"f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c"} Oct 30 00:48:09.478126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:09.478085 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" podStartSLOduration=1.4780723550000001 podStartE2EDuration="1.478072355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:09.475403811 +0000 UTC m=+941.715638926" watchObservedRunningTime="2026-10-30 00:48:09.478072355 +0000 UTC m=+941.718307467" Oct 30 00:48:10.445940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:10.445898 5938 generic.go:356] "Generic (PLEG): container finished" podID="9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" containerID="0e671fd6632b0759cc7b4b86195d9b036dfd3d77a467912bbb91ffe61208bbe8" exitCode=0 Oct 30 00:48:10.446380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:10.446017 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" event={"ID":"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d","Type":"ContainerDied","Data":"0e671fd6632b0759cc7b4b86195d9b036dfd3d77a467912bbb91ffe61208bbe8"} Oct 30 00:48:10.870002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:10.869972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:11.615969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.615949 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:11.684838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.684808 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\") pod \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " Oct 30 00:48:11.685020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.684941 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\") pod \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " Oct 30 00:48:11.685152 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.685137 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\" (UniqueName: \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\") pod \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\" (UID: \"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d\") " Oct 30 00:48:11.685313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.685284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca" pod "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" (UID: "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:11.685697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.685674 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:11.687111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.687058 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca" pod "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" (UID: "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:11.687463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.687437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw" pod "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" (UID: "9122261f-ea39-4ed3-834c-a4e0c7a2aa0d"). InnerVolumeSpecName "kube-api-access-fc8fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:11.787400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.787345 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc8fw\" (UniqueName: \"kubernetes.io/projected/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-kube-api-access-fc8fw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:11.787400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:11.787367 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:12.461671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:12.461642 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" Oct 30 00:48:12.461858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:12.461681 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql" event={"ID":"9122261f-ea39-4ed3-834c-a4e0c7a2aa0d","Type":"ContainerDied","Data":"f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c"} Oct 30 00:48:12.461858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:12.461738 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4dcf6238fb21e8a7f3877621d3a89ab4150ba1db2c88653b0833f42c477cc9c" Oct 30 00:48:12.477692 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:12.477662 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:13.835295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.835261 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7"] Oct 30 00:48:13.835834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.835818 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" containerName="revoke-keys" Oct 30 00:48:13.835834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.835834 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" containerName="revoke-keys" Oct 30 00:48:13.835992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.835980 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" containerName="revoke-keys" Oct 30 00:48:13.839741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.839722 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:13.868064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.868037 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7"] Oct 30 00:48:13.873069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.873040 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:13.873226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.873186 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:13.873294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.873251 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:13.873350 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.873327 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:13.914917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.914887 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:13.915031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.914991 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:13.915175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.915162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975bt\" (UniqueName: \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:13.969642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:13.969618 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.016952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.016910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975bt\" (UniqueName: \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.017165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.017091 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.017245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.017164 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.018027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.017978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.020498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.020468 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.040196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.040167 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975bt\" (UniqueName: \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\") pod \"maas-api-revoke-sub-6da53c58-z44k7\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.161841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.161807 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:14.463288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.463259 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7"] Oct 30 00:48:14.463857 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:14.463828 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa2fc28_76d9_4e4e_bbed_7fd216cc6270.slice/crio-7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2 WatchSource:0}: Error finding container 7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2: Status 404 returned error can't find the container with id 7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2 Oct 30 00:48:14.478559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:14.478529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" event={"ID":"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270","Type":"ContainerStarted","Data":"7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2"} Oct 30 00:48:15.487113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:15.487079 5938 generic.go:356] "Generic (PLEG): container finished" podID="4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" containerID="f44bc9a1dd147f2735defad66a413ae4ec376e35bf25b7e851e694002833c287" exitCode=0 Oct 30 00:48:15.487453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:15.487161 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" event={"ID":"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270","Type":"ContainerDied","Data":"f44bc9a1dd147f2735defad66a413ae4ec376e35bf25b7e851e694002833c287"} Oct 30 00:48:15.540565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:15.540540 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:16.668047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.668024 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:16.749373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.749303 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\" (UniqueName: \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\") pod \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " Oct 30 00:48:16.749521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.749414 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\") pod \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " Oct 30 00:48:16.749592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.749537 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\") pod \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\" (UID: \"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270\") " Oct 30 00:48:16.750108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.750052 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca" pod "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" (UID: "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:16.750264 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.750181 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:16.751930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.751906 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca" pod "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" (UID: "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:16.752035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.751977 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt" pod "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" (UID: "4fa2fc28-76d9-4e4e-bbed-7fd216cc6270"). InnerVolumeSpecName "kube-api-access-975bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:16.851341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.851319 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975bt\" (UniqueName: \"kubernetes.io/projected/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-kube-api-access-975bt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:16.851341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:16.851341 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:17.138774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:17.138733 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:17.503676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:17.503592 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" Oct 30 00:48:17.503676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:17.503622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7" event={"ID":"4fa2fc28-76d9-4e4e-bbed-7fd216cc6270","Type":"ContainerDied","Data":"7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2"} Oct 30 00:48:17.503948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:17.503701 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e309798e29ae053415fa4a5f792105cefac756b324a36614806553a1e5fcab2" Oct 30 00:48:18.412151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.412120 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5"] Oct 30 00:48:18.412726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.412712 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" containerName="revoke-keys" Oct 30 00:48:18.412779 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.412727 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" containerName="revoke-keys" Oct 30 00:48:18.412846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.412838 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" containerName="revoke-keys" Oct 30 00:48:18.415986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.415971 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.430030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.430005 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:18.430030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.430010 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:18.433981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.433958 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5"] Oct 30 00:48:18.440453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.440420 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:18.440453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.440445 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:18.572965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.572935 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.573142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.573090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchzx\" (UniqueName: \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.573215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.573198 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.674584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.674515 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchzx\" (UniqueName: \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.674724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.674592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.674724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.674671 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.675311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.675287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.676982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.676957 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.697017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.696994 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchzx\" (UniqueName: \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\") pod \"maas-api-revoke-sub-ea490f89-c6sw5\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:18.728789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:18.728767 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:19.036542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:19.036510 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5"] Oct 30 00:48:19.037122 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:19.037099 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37b3109_962c_445c_8120_413dc06d2a60.slice/crio-a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b WatchSource:0}: Error finding container a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b: Status 404 returned error can't find the container with id a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b Oct 30 00:48:19.178304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:19.178272 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:19.519547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:19.519504 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" event={"ID":"b37b3109-962c-445c-8120-413dc06d2a60","Type":"ContainerStarted","Data":"9f725dc337c42a624e998494a3e76cb7799801693789f8a915d6ae8b2a8d9604"} Oct 30 00:48:19.520030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:19.519563 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" event={"ID":"b37b3109-962c-445c-8120-413dc06d2a60","Type":"ContainerStarted","Data":"a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b"} Oct 30 00:48:19.558389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:19.558310 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" podStartSLOduration=1.558297279 podStartE2EDuration="1.558297279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:19.557119607 +0000 UTC m=+951.797354721" watchObservedRunningTime="2026-10-30 00:48:19.558297279 +0000 UTC m=+951.798532392" Oct 30 00:48:20.469735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:20.469678 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:20.530251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:20.530215 5938 generic.go:356] "Generic (PLEG): container finished" podID="b37b3109-962c-445c-8120-413dc06d2a60" containerID="9f725dc337c42a624e998494a3e76cb7799801693789f8a915d6ae8b2a8d9604" exitCode=0 Oct 30 00:48:20.530743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:20.530294 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" event={"ID":"b37b3109-962c-445c-8120-413dc06d2a60","Type":"ContainerDied","Data":"9f725dc337c42a624e998494a3e76cb7799801693789f8a915d6ae8b2a8d9604"} Oct 30 00:48:21.098154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.098111 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:48:21.104007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.103968 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.129423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.129366 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267301 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267384 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267420 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267526 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267591 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267658 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflfn\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267696 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.269937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.267736 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.454802 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.454923 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455115 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455151 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455180 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455270 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455347 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.457891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.455420 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflfn\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.540886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.489662 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.540886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.510332 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.540886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.510720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.540886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.511064 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.540886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.519934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.782030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.764646 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.821616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.821584 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflfn\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.851171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.838243 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:21.851171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:21.851116 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:22.029446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.029411 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:22.324934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.311960 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:22.648886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.648476 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.058807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.983980 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.058807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.985302 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.058807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:22.998226 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:48:23.058807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.009283 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Oct 30 00:48:23.095903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.082999 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.221272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.220765 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:48:23.263023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.237945 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:48:23.263023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.238192 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:23.263023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.238692 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.263023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.238892 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.391194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.323138 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.391194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.323925 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:23.655930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.652288 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:23.655930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.654082 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:23.655930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.655026 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:23.655930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:23.655065 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.040802 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.041191 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.042142 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.056966 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.057208 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.065399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.057691 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.253998 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.256494 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.257472 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.262225 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.262265 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.269717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.262326 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:48:24.282898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.277020 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:48:24.359986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.323993 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.442893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.441893 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:24.442893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.441993 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:24.442893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.442037 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:24.563078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.563038 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.654969 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.655316 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658433 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658479 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658582 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658601 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" event={"ID":"7a9c2842-ed5c-4bf1-a9b2-980dca377beb","Type":"ContainerStarted","Data":"9f247cafb3c0a05711d1ab4553eafed2a042aa7f0c77fb1e8f6d9613f9f9559f"} Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" event={"ID":"b37b3109-962c-445c-8120-413dc06d2a60","Type":"ContainerDied","Data":"a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b"} Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658658 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20c31b455b4173241941e1cf9cbc26b71980b45928aaab874ca3e12f5862d2b" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658698 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658739 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658774 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658804 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:24.684231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.658837 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:24.685673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:24.685643 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:25.111826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.111782 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:48:25.120600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.120558 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:48:25.142395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.132464 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.254311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.254271 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.300348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.300292 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" podUID="671141fa-f7fe-4f3f-952e-bf6aebb07516" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.19:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.313279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.305521 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.399965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.375103 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:48:25.400430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.392437 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.400789 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.392889 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:48:25.401116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.398609 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:48:25.817365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.816586 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:25.838887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.819968 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:25.844179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.844159 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:25.846437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.845526 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:25.855111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.855080 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:25.861683 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.861036 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:25.861683 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.861426 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:25.862985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.862569 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:25.877879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.866289 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\" (UniqueName: \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\") pod \"b37b3109-962c-445c-8120-413dc06d2a60\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " Oct 30 00:48:25.880022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.879990 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\") pod \"b37b3109-962c-445c-8120-413dc06d2a60\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " Oct 30 00:48:25.880123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.880063 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\") pod \"b37b3109-962c-445c-8120-413dc06d2a60\" (UID: \"b37b3109-962c-445c-8120-413dc06d2a60\") " Oct 30 00:48:25.883775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.883729 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca" pod "b37b3109-962c-445c-8120-413dc06d2a60" (UID: "b37b3109-962c-445c-8120-413dc06d2a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:25.894857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.886908 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5" Oct 30 00:48:25.901274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.900436 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca" pod "b37b3109-962c-445c-8120-413dc06d2a60" (UID: "b37b3109-962c-445c-8120-413dc06d2a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:25.917679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.912460 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:25.940930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.929065 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:25.940930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.929231 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:25.940930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.934466 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:25.940930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.937621 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx" pod "b37b3109-962c-445c-8120-413dc06d2a60" (UID: "b37b3109-962c-445c-8120-413dc06d2a60"). InnerVolumeSpecName "kube-api-access-vchzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:25.982883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.982826 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:25.983126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.983105 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37b3109-962c-445c-8120-413dc06d2a60-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:25.983275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:25.983259 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vchzx\" (UniqueName: \"kubernetes.io/projected/b37b3109-962c-445c-8120-413dc06d2a60-kube-api-access-vchzx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:26.215776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.215741 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk"] Oct 30 00:48:26.216699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.216673 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b37b3109-962c-445c-8120-413dc06d2a60" containerName="revoke-keys" Oct 30 00:48:26.216699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.216698 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37b3109-962c-445c-8120-413dc06d2a60" containerName="revoke-keys" Oct 30 00:48:26.216920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.216897 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37b3109-962c-445c-8120-413dc06d2a60" containerName="revoke-keys" Oct 30 00:48:26.220942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.220919 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.235458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.235430 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:26.235624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.235505 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:26.235697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.235639 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:26.235767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.235728 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:26.240981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.240888 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk"] Oct 30 00:48:26.296838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.296801 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjmv\" (UniqueName: \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.297025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.296980 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.297110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.297045 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.398727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.398690 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjmv\" (UniqueName: \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.398966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.398894 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.398966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.398950 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.399993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.399967 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.401617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.401572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.424083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.424029 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjmv\" (UniqueName: \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\") pod \"maas-api-revoke-sub-cf145eca-lv4xk\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.520258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.520172 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42"] Oct 30 00:48:26.525592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.525565 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.579993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.579953 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42"] Oct 30 00:48:26.618040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.618002 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:26.625009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.623181 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.625009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.623349 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.625009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.623502 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzf24\" (UniqueName: \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.725639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.725596 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.725774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.725686 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzf24\" (UniqueName: \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.727249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.726586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.727249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.726811 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.731285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.731258 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.749272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.749242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzf24\" (UniqueName: \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\") pod \"maas-api-revoke-sub-3d862376-w8c42\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.913889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.913561 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" event={"ID":"7a9c2842-ed5c-4bf1-a9b2-980dca377beb","Type":"ContainerStarted","Data":"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325"} Oct 30 00:48:26.921260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.921233 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:26.958839 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:26.958674 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8614d5_d852_428e_9489_748d141cf31f.slice/crio-dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde WatchSource:0}: Error finding container dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde: Status 404 returned error can't find the container with id dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde Oct 30 00:48:26.960577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.960559 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk"] Oct 30 00:48:26.962710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:26.962666 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podStartSLOduration=5.962652213 podStartE2EDuration="5.962652213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:26.96021451 +0000 UTC m=+959.200449629" watchObservedRunningTime="2026-10-30 00:48:26.962652213 +0000 UTC m=+959.202887327" Oct 30 00:48:27.031413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.030686 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:27.032535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.031938 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:27.032535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.031995 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:27.032535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.032272 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:27.254700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.254668 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42"] Oct 30 00:48:27.255064 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:27.255044 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c739bb_7dfa_4064_bbfd_826e0ebd4e16.slice/crio-21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a WatchSource:0}: Error finding container 21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a: Status 404 returned error can't find the container with id 21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a Oct 30 00:48:27.923168 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.923136 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" event={"ID":"0c8614d5-d852-428e-9489-748d141cf31f","Type":"ContainerStarted","Data":"d5219b6404a023b1658a397e58ea603c16fbf70aae3523e12ecc4680864108b9"} Oct 30 00:48:27.923633 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.923194 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" event={"ID":"0c8614d5-d852-428e-9489-748d141cf31f","Type":"ContainerStarted","Data":"dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde"} Oct 30 00:48:27.925017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.924986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" event={"ID":"97c739bb-7dfa-4064-bbfd-826e0ebd4e16","Type":"ContainerStarted","Data":"c57a9da34d02b3fd655e090bca6606c6113fc503a51b346b29ace3e7d438736f"} Oct 30 00:48:27.925142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.925041 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" event={"ID":"97c739bb-7dfa-4064-bbfd-826e0ebd4e16","Type":"ContainerStarted","Data":"21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a"} Oct 30 00:48:27.964639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:27.964577 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" podStartSLOduration=1.964563665 podStartE2EDuration="1.964563665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:27.962507302 +0000 UTC m=+960.202742444" watchObservedRunningTime="2026-10-30 00:48:27.964563665 +0000 UTC m=+960.204798778" Oct 30 00:48:28.015813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.015760 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" podStartSLOduration=2.015746257 podStartE2EDuration="2.015746257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:28.013218858 +0000 UTC m=+960.253453972" watchObservedRunningTime="2026-10-30 00:48:28.015746257 +0000 UTC m=+960.255981364" Oct 30 00:48:28.030614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.030585 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:28.030737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.030637 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:28.758421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.758387 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:28.883554 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:48:28.883513 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:48:28.934708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.934673 5938 generic.go:356] "Generic (PLEG): container finished" podID="97c739bb-7dfa-4064-bbfd-826e0ebd4e16" containerID="c57a9da34d02b3fd655e090bca6606c6113fc503a51b346b29ace3e7d438736f" exitCode=0 Oct 30 00:48:28.935116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.934752 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" event={"ID":"97c739bb-7dfa-4064-bbfd-826e0ebd4e16","Type":"ContainerDied","Data":"c57a9da34d02b3fd655e090bca6606c6113fc503a51b346b29ace3e7d438736f"} Oct 30 00:48:28.936768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.936743 5938 generic.go:356] "Generic (PLEG): container finished" podID="0c8614d5-d852-428e-9489-748d141cf31f" containerID="d5219b6404a023b1658a397e58ea603c16fbf70aae3523e12ecc4680864108b9" exitCode=0 Oct 30 00:48:28.936902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:28.936809 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" event={"ID":"0c8614d5-d852-428e-9489-748d141cf31f","Type":"ContainerDied","Data":"d5219b6404a023b1658a397e58ea603c16fbf70aae3523e12ecc4680864108b9"} Oct 30 00:48:29.030760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:29.030688 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:29.030964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:29.030754 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:30.036748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.031278 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:30.036748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.031362 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:30.156635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.156608 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:30.226418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.226356 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294386 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\") pod \"0c8614d5-d852-428e-9489-748d141cf31f\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294444 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\") pod \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294485 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\" (UniqueName: \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\") pod \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294621 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\") pod \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\" (UID: \"97c739bb-7dfa-4064-bbfd-826e0ebd4e16\") " Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294662 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\") pod \"0c8614d5-d852-428e-9489-748d141cf31f\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " Oct 30 00:48:30.301887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.294759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\" (UniqueName: \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\") pod \"0c8614d5-d852-428e-9489-748d141cf31f\" (UID: \"0c8614d5-d852-428e-9489-748d141cf31f\") " Oct 30 00:48:30.315970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.315940 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca" pod "97c739bb-7dfa-4064-bbfd-826e0ebd4e16" (UID: "97c739bb-7dfa-4064-bbfd-826e0ebd4e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:30.316383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.316354 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.316383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.316350 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca" pod "0c8614d5-d852-428e-9489-748d141cf31f" (UID: "0c8614d5-d852-428e-9489-748d141cf31f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:30.318407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.318384 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24" pod "97c739bb-7dfa-4064-bbfd-826e0ebd4e16" (UID: "97c739bb-7dfa-4064-bbfd-826e0ebd4e16"). InnerVolumeSpecName "kube-api-access-pzf24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:30.323649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.320522 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv" pod "0c8614d5-d852-428e-9489-748d141cf31f" (UID: "0c8614d5-d852-428e-9489-748d141cf31f"). InnerVolumeSpecName "kube-api-access-7hjmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:30.323649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.320696 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca" pod "97c739bb-7dfa-4064-bbfd-826e0ebd4e16" (UID: "97c739bb-7dfa-4064-bbfd-826e0ebd4e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:30.323649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.321041 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca" pod "0c8614d5-d852-428e-9489-748d141cf31f" (UID: "0c8614d5-d852-428e-9489-748d141cf31f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:30.333952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.333015 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:30.418482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.418303 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.418482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.418336 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.418482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.418351 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzf24\" (UniqueName: \"kubernetes.io/projected/97c739bb-7dfa-4064-bbfd-826e0ebd4e16-kube-api-access-pzf24\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.418482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.418364 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c8614d5-d852-428e-9489-748d141cf31f-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.418482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.418377 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjmv\" (UniqueName: \"kubernetes.io/projected/0c8614d5-d852-428e-9489-748d141cf31f-kube-api-access-7hjmv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:30.536036 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.535960 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4"] Oct 30 00:48:30.536548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536535 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0c8614d5-d852-428e-9489-748d141cf31f" containerName="revoke-keys" Oct 30 00:48:30.536586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536550 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8614d5-d852-428e-9489-748d141cf31f" containerName="revoke-keys" Oct 30 00:48:30.536586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536583 5938 cpu_manager.go:414] "containerMap: removing container" podUID="97c739bb-7dfa-4064-bbfd-826e0ebd4e16" containerName="revoke-keys" Oct 30 00:48:30.536647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536589 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c739bb-7dfa-4064-bbfd-826e0ebd4e16" containerName="revoke-keys" Oct 30 00:48:30.536696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536687 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8614d5-d852-428e-9489-748d141cf31f" containerName="revoke-keys" Oct 30 00:48:30.536735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.536705 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c739bb-7dfa-4064-bbfd-826e0ebd4e16" containerName="revoke-keys" Oct 30 00:48:30.540565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.540546 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.558116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.558089 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4"] Oct 30 00:48:30.620919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.620888 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.621078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.621061 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znv9j\" (UniqueName: \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.621126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.621109 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.722531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.722503 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.722671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.722625 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znv9j\" (UniqueName: \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.722671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.722658 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.723245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.723222 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.725028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.725008 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.744987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.744966 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znv9j\" (UniqueName: \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\") pod \"maas-api-revoke-sub-2ae9b907-mnpv4\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.854197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.854175 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:30.956585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.956549 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" event={"ID":"0c8614d5-d852-428e-9489-748d141cf31f","Type":"ContainerDied","Data":"dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde"} Oct 30 00:48:30.956704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.956606 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dae09e043d367312afa578728bb41d3f95edc5828fe7c4e7112a8be1d3403cde" Oct 30 00:48:30.956704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.956658 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk" Oct 30 00:48:30.958333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.958311 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" event={"ID":"97c739bb-7dfa-4064-bbfd-826e0ebd4e16","Type":"ContainerDied","Data":"21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a"} Oct 30 00:48:30.958469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.958459 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f6f175de5e0da00b57af99d2c318c62eff5cce33360f3b5f8f7812cf48fb9a" Oct 30 00:48:30.958592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:30.958583 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42" Oct 30 00:48:31.030250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.030222 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:31.030401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.030269 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:31.071289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.071266 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:31.178403 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:31.178368 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89beefc6_8952_4c5b_84c5_81142d45353a.slice/crio-8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90 WatchSource:0}: Error finding container 8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90: Status 404 returned error can't find the container with id 8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90 Oct 30 00:48:31.178713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.178694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4"] Oct 30 00:48:31.967600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.967513 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" event={"ID":"89beefc6-8952-4c5b-84c5-81142d45353a","Type":"ContainerStarted","Data":"4cab00f107e45e5911c9287699d50984c6ce3be31c3e1df0fcfaf77c5fa0bac5"} Oct 30 00:48:31.967600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:31.967572 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" event={"ID":"89beefc6-8952-4c5b-84c5-81142d45353a","Type":"ContainerStarted","Data":"8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90"} Oct 30 00:48:32.009276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.009222 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" podStartSLOduration=2.009208935 podStartE2EDuration="2.009208935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:32.005690702 +0000 UTC m=+964.245925827" watchObservedRunningTime="2026-10-30 00:48:32.009208935 +0000 UTC m=+964.249444048" Oct 30 00:48:32.031185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.031154 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 00:48:32.031346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.031205 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 00:48:32.068345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.068306 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:32.980784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.980743 5938 generic.go:356] "Generic (PLEG): container finished" podID="89beefc6-8952-4c5b-84c5-81142d45353a" containerID="4cab00f107e45e5911c9287699d50984c6ce3be31c3e1df0fcfaf77c5fa0bac5" exitCode=0 Oct 30 00:48:32.981221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:32.980800 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" event={"ID":"89beefc6-8952-4c5b-84c5-81142d45353a","Type":"ContainerDied","Data":"4cab00f107e45e5911c9287699d50984c6ce3be31c3e1df0fcfaf77c5fa0bac5"} Oct 30 00:48:33.380718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:33.379699 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:33.380718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:33.380130 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:33.381079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:33.381054 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:48:33.619567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:33.619528 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:34.155670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.155649 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:34.279779 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.279706 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\") pod \"89beefc6-8952-4c5b-84c5-81142d45353a\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " Oct 30 00:48:34.279946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.279799 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\" (UniqueName: \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\") pod \"89beefc6-8952-4c5b-84c5-81142d45353a\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " Oct 30 00:48:34.280045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.280028 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\") pod \"89beefc6-8952-4c5b-84c5-81142d45353a\" (UID: \"89beefc6-8952-4c5b-84c5-81142d45353a\") " Oct 30 00:48:34.280094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.280049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca" pod "89beefc6-8952-4c5b-84c5-81142d45353a" (UID: "89beefc6-8952-4c5b-84c5-81142d45353a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:34.280604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.280582 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:34.281911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.281887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j" pod "89beefc6-8952-4c5b-84c5-81142d45353a" (UID: "89beefc6-8952-4c5b-84c5-81142d45353a"). InnerVolumeSpecName "kube-api-access-znv9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:34.281983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.281941 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca" pod "89beefc6-8952-4c5b-84c5-81142d45353a" (UID: "89beefc6-8952-4c5b-84c5-81142d45353a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:34.381599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.381576 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89beefc6-8952-4c5b-84c5-81142d45353a-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:34.381599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.381598 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znv9j\" (UniqueName: \"kubernetes.io/projected/89beefc6-8952-4c5b-84c5-81142d45353a-kube-api-access-znv9j\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:34.997301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.997272 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" Oct 30 00:48:34.997486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.997297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4" event={"ID":"89beefc6-8952-4c5b-84c5-81142d45353a","Type":"ContainerDied","Data":"8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90"} Oct 30 00:48:34.997486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:34.997349 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8822050cb6bde2c3c402df2e876793b06369a41d456e58a9a785cd04d7b42e90" Oct 30 00:48:35.127197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:35.127169 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:35.938149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:35.938117 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:36.790440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:36.790404 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:37.228162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.228129 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6"] Oct 30 00:48:37.228690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.228675 5938 cpu_manager.go:414] "containerMap: removing container" podUID="89beefc6-8952-4c5b-84c5-81142d45353a" containerName="revoke-keys" Oct 30 00:48:37.228731 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.228690 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89beefc6-8952-4c5b-84c5-81142d45353a" containerName="revoke-keys" Oct 30 00:48:37.228827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.228817 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="89beefc6-8952-4c5b-84c5-81142d45353a" containerName="revoke-keys" Oct 30 00:48:37.232495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.232473 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.250291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.250268 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6"] Oct 30 00:48:37.256751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.256726 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:37.256895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.256765 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:37.258325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.258305 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:37.258444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.258373 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:37.319557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.319530 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkpq\" (UniqueName: \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.319677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.319570 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.319677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.319646 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.421027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.421002 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.421179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.421166 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkpq\" (UniqueName: \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.421225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.421194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.421636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.421616 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.423528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.423503 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.443708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.443680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkpq\" (UniqueName: \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\") pod \"maas-api-revoke-sub-ceadfc08-zscr6\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.546799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.546734 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:37.856337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:37.856305 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6"] Oct 30 00:48:37.856536 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:37.856509 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb437c0b7_95ff_486b_bec0_f07d6d73b792.slice/crio-e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01 WatchSource:0}: Error finding container e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01: Status 404 returned error can't find the container with id e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01 Oct 30 00:48:38.025535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:38.025501 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" event={"ID":"b437c0b7-95ff-486b-bec0-f07d6d73b792","Type":"ContainerStarted","Data":"e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01"} Oct 30 00:48:38.687936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:38.687894 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:39.035188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.035115 5938 generic.go:356] "Generic (PLEG): container finished" podID="b437c0b7-95ff-486b-bec0-f07d6d73b792" containerID="807805be4605c00d38fd39d318072eb8e5dd5d0a13e3bb43e55ab45209fd0fee" exitCode=0 Oct 30 00:48:39.035188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.035154 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" event={"ID":"b437c0b7-95ff-486b-bec0-f07d6d73b792","Type":"ContainerDied","Data":"807805be4605c00d38fd39d318072eb8e5dd5d0a13e3bb43e55ab45209fd0fee"} Oct 30 00:48:39.325923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.325835 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk"] Oct 30 00:48:39.331947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.331925 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.347437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.347410 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk"] Oct 30 00:48:39.450001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.449966 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.450203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.450183 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.450272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.450259 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465br\" (UniqueName: \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.551379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.551352 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.551502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.551411 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465br\" (UniqueName: \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.551502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.551494 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.552053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.552022 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.553781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.553759 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.573582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.573557 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465br\" (UniqueName: \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\") pod \"maas-api-revoke-sub-1b99671f-bwlxk\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.648975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.648954 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:39.949675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:39.949648 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk"] Oct 30 00:48:39.951573 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:39.951546 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3424d7_a7f1_41c1_9c3e_80d2580fe041.slice/crio-2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc WatchSource:0}: Error finding container 2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc: Status 404 returned error can't find the container with id 2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc Oct 30 00:48:40.044321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.044294 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" event={"ID":"2e3424d7-a7f1-41c1-9c3e-80d2580fe041","Type":"ContainerStarted","Data":"2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc"} Oct 30 00:48:40.159115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.159079 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:40.218410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.218378 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:40.262023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.261496 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\" (UniqueName: \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\") pod \"b437c0b7-95ff-486b-bec0-f07d6d73b792\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " Oct 30 00:48:40.262023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.261632 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\") pod \"b437c0b7-95ff-486b-bec0-f07d6d73b792\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " Oct 30 00:48:40.262023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.261735 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\") pod \"b437c0b7-95ff-486b-bec0-f07d6d73b792\" (UID: \"b437c0b7-95ff-486b-bec0-f07d6d73b792\") " Oct 30 00:48:40.262870 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.262839 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca" pod "b437c0b7-95ff-486b-bec0-f07d6d73b792" (UID: "b437c0b7-95ff-486b-bec0-f07d6d73b792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:40.270506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.269587 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq" pod "b437c0b7-95ff-486b-bec0-f07d6d73b792" (UID: "b437c0b7-95ff-486b-bec0-f07d6d73b792"). InnerVolumeSpecName "kube-api-access-crkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:40.274480 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.270901 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca" pod "b437c0b7-95ff-486b-bec0-f07d6d73b792" (UID: "b437c0b7-95ff-486b-bec0-f07d6d73b792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:40.363942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.363913 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkpq\" (UniqueName: \"kubernetes.io/projected/b437c0b7-95ff-486b-bec0-f07d6d73b792-kube-api-access-crkpq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:40.363942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.363937 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:40.363942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.363948 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b437c0b7-95ff-486b-bec0-f07d6d73b792-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:40.665020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.664991 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg"] Oct 30 00:48:40.665784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.665755 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b437c0b7-95ff-486b-bec0-f07d6d73b792" containerName="revoke-keys" Oct 30 00:48:40.665784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.665780 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b437c0b7-95ff-486b-bec0-f07d6d73b792" containerName="revoke-keys" Oct 30 00:48:40.665995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.665942 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b437c0b7-95ff-486b-bec0-f07d6d73b792" containerName="revoke-keys" Oct 30 00:48:40.669716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.669699 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.688044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.688018 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg"] Oct 30 00:48:40.769394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.769362 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.769559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.769542 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.769597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.769581 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkf6\" (UniqueName: \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.870327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.870302 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.870479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.870344 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkf6\" (UniqueName: \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.870479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.870439 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.870989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.870966 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.872711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.872689 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.892595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.892572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkf6\" (UniqueName: \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\") pod \"maas-api-revoke-sub-6c5dca60-4m4gg\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:40.983480 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:40.983457 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:41.061367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.061328 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" event={"ID":"2e3424d7-a7f1-41c1-9c3e-80d2580fe041","Type":"ContainerStarted","Data":"a4cddd656ff21e8c099fe4012a3e1644af527e1e8c17c856ab4c2e0461022c95"} Oct 30 00:48:41.063297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.063265 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" event={"ID":"b437c0b7-95ff-486b-bec0-f07d6d73b792","Type":"ContainerDied","Data":"e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01"} Oct 30 00:48:41.063297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.063277 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6" Oct 30 00:48:41.063457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.063307 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e549a09f9064ce49254f4fd07577ce27b8e2801bc38bdd157d298172f4811a01" Oct 30 00:48:41.101915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.101834 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" podStartSLOduration=2.101818317 podStartE2EDuration="2.101818317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:41.100747224 +0000 UTC m=+973.340982340" watchObservedRunningTime="2026-10-30 00:48:41.101818317 +0000 UTC m=+973.342053432" Oct 30 00:48:41.287549 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:41.287516 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646635fd_4d84_42a1_972e_71312c047d3c.slice/crio-b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d WatchSource:0}: Error finding container b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d: Status 404 returned error can't find the container with id b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d Oct 30 00:48:41.289815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.289791 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg"] Oct 30 00:48:41.753280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:41.753256 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:42.072630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:42.072594 5938 generic.go:356] "Generic (PLEG): container finished" podID="2e3424d7-a7f1-41c1-9c3e-80d2580fe041" containerID="a4cddd656ff21e8c099fe4012a3e1644af527e1e8c17c856ab4c2e0461022c95" exitCode=0 Oct 30 00:48:42.073071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:42.072678 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" event={"ID":"2e3424d7-a7f1-41c1-9c3e-80d2580fe041","Type":"ContainerDied","Data":"a4cddd656ff21e8c099fe4012a3e1644af527e1e8c17c856ab4c2e0461022c95"} Oct 30 00:48:42.074454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:42.074430 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" event={"ID":"646635fd-4d84-42a1-972e-71312c047d3c","Type":"ContainerStarted","Data":"c81b43c2dd6e3ac30d6639329cfbcea3a638a2c1f166d5166577b163544fbbfa"} Oct 30 00:48:42.074553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:42.074471 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" event={"ID":"646635fd-4d84-42a1-972e-71312c047d3c","Type":"ContainerStarted","Data":"b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d"} Oct 30 00:48:42.152478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:42.152430 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" podStartSLOduration=2.152416123 podStartE2EDuration="2.152416123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:42.15153206 +0000 UTC m=+974.391767176" watchObservedRunningTime="2026-10-30 00:48:42.152416123 +0000 UTC m=+974.392651236" Oct 30 00:48:43.084730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.084697 5938 generic.go:356] "Generic (PLEG): container finished" podID="646635fd-4d84-42a1-972e-71312c047d3c" containerID="c81b43c2dd6e3ac30d6639329cfbcea3a638a2c1f166d5166577b163544fbbfa" exitCode=0 Oct 30 00:48:43.087308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.084732 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" event={"ID":"646635fd-4d84-42a1-972e-71312c047d3c","Type":"ContainerDied","Data":"c81b43c2dd6e3ac30d6639329cfbcea3a638a2c1f166d5166577b163544fbbfa"} Oct 30 00:48:43.272924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.272896 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:43.316472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.316442 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:43.414292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.413052 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\" (UniqueName: \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\") pod \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " Oct 30 00:48:43.414292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.413351 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\") pod \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " Oct 30 00:48:43.414292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.413427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\") pod \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\" (UID: \"2e3424d7-a7f1-41c1-9c3e-80d2580fe041\") " Oct 30 00:48:43.429887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.424104 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca" pod "2e3424d7-a7f1-41c1-9c3e-80d2580fe041" (UID: "2e3424d7-a7f1-41c1-9c3e-80d2580fe041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:43.439538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.435283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca" pod "2e3424d7-a7f1-41c1-9c3e-80d2580fe041" (UID: "2e3424d7-a7f1-41c1-9c3e-80d2580fe041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:43.455923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.455853 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br" pod "2e3424d7-a7f1-41c1-9c3e-80d2580fe041" (UID: "2e3424d7-a7f1-41c1-9c3e-80d2580fe041"). InnerVolumeSpecName "kube-api-access-465br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:43.483097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.482780 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:43.516365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.516287 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:43.516365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.516320 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:43.516365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:43.516336 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465br\" (UniqueName: \"kubernetes.io/projected/2e3424d7-a7f1-41c1-9c3e-80d2580fe041-kube-api-access-465br\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:44.177794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:44.177707 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" Oct 30 00:48:44.179077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:44.179048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk" event={"ID":"2e3424d7-a7f1-41c1-9c3e-80d2580fe041","Type":"ContainerDied","Data":"2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc"} Oct 30 00:48:44.179245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:44.179231 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c344af5ba9bc8e8f2c42a932a8d90aeb53535bc4efeca7a992a443f307014fc" Oct 30 00:48:45.289894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.271917 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:45.404620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.402836 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:45.404620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.403804 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:45.601946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.576144 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:45.742890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.739371 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\" (UniqueName: \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\") pod \"646635fd-4d84-42a1-972e-71312c047d3c\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " Oct 30 00:48:45.742890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.739564 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\") pod \"646635fd-4d84-42a1-972e-71312c047d3c\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " Oct 30 00:48:45.742890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.739759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\") pod \"646635fd-4d84-42a1-972e-71312c047d3c\" (UID: \"646635fd-4d84-42a1-972e-71312c047d3c\") " Oct 30 00:48:45.767888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.754688 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca" pod "646635fd-4d84-42a1-972e-71312c047d3c" (UID: "646635fd-4d84-42a1-972e-71312c047d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:45.964893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.946457 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:46.162888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.969406 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.162888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:45.991965 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.162888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.086735 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.179765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.108514 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.208108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.190031 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6" pod "646635fd-4d84-42a1-972e-71312c047d3c" (UID: "646635fd-4d84-42a1-972e-71312c047d3c"). InnerVolumeSpecName "kube-api-access-6mkf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:46.208108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.193970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca" pod "646635fd-4d84-42a1-972e-71312c047d3c" (UID: "646635fd-4d84-42a1-972e-71312c047d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:46.208108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.201227 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:48:46.265892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.265807 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mkf6\" (UniqueName: \"kubernetes.io/projected/646635fd-4d84-42a1-972e-71312c047d3c-kube-api-access-6mkf6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:46.265892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.265846 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646635fd-4d84-42a1-972e-71312c047d3c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:46.287886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.278301 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:48:46.287886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.278374 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.287886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.278661 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.287886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.279769 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:46.410113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.410072 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.411820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.410755 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.434933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.412116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:46.434933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.412173 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:46.734359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.724103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:46.933899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.917938 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:48:46.994940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.977102 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:46.997850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:46.997753 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:47.203502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.179732 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:48:47.203502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.180993 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:47.283941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.210002 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:48:47.291887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.216275 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:47.291887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.287259 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:47.291887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.216530 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:48:47.316886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.309012 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:47.338484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.327440 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" Oct 30 00:48:47.374463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.374428 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg" event={"ID":"646635fd-4d84-42a1-972e-71312c047d3c","Type":"ContainerDied","Data":"b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d"} Oct 30 00:48:47.374687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.374667 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b445b856212af03f632a4bb97757482368d70264d9d867d46bb2f833acd8260d" Oct 30 00:48:47.374894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.374877 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:47.375085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.375070 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:47.412015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.411976 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:47.412368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.412342 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:47.921888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:47.921145 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.035650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.035607 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:48.059329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.059288 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:48:48.204588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.204481 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.290209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.290165 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.290790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.290474 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.530499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.290507 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.541924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.316722 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.542825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.402627 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:48.560156 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.490333 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:48:48.561732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.490368 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:48:48.677470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.677437 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:48.680787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.680766 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:48.681613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.681595 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:48.716664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.716636 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:48:48.801120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.801040 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:48:48.802319 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.754905 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:48.928062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:48.928023 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:49.065273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.065183 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:49.065809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.065775 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:48:49.100169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.100138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:48:49.144324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.144296 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:48:49.218275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.201491 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:48:49.218275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.206938 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:48:49.223038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.222332 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:48:49.266942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.255204 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:48:49.266942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.256103 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:48:49.271821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.270738 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:48:49.284297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.284273 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:48:49.675826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.675782 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q"] Oct 30 00:48:49.676673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676619 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2e3424d7-a7f1-41c1-9c3e-80d2580fe041" containerName="revoke-keys" Oct 30 00:48:49.676673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676645 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3424d7-a7f1-41c1-9c3e-80d2580fe041" containerName="revoke-keys" Oct 30 00:48:49.676848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676697 5938 cpu_manager.go:414] "containerMap: removing container" podUID="646635fd-4d84-42a1-972e-71312c047d3c" containerName="revoke-keys" Oct 30 00:48:49.676848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676705 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646635fd-4d84-42a1-972e-71312c047d3c" containerName="revoke-keys" Oct 30 00:48:49.677004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676964 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="646635fd-4d84-42a1-972e-71312c047d3c" containerName="revoke-keys" Oct 30 00:48:49.677004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.676990 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3424d7-a7f1-41c1-9c3e-80d2580fe041" containerName="revoke-keys" Oct 30 00:48:49.680790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.680769 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.697912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.696209 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:48:49.697912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.696391 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:48:49.697912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.696420 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:48:49.697912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.696503 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:48:49.698834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.698810 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q"] Oct 30 00:48:49.713433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.713405 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngt5m\" (UniqueName: \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.713607 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.713585 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.713680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.713620 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.815455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.815426 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngt5m\" (UniqueName: \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.815640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.815618 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.815793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.815778 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.816364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.816341 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.818037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.818019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.837636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.837611 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngt5m\" (UniqueName: \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\") pod \"maas-api-revoke-sub-948ad435-dhv4q\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:49.995253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:49.995178 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:50.043365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.043332 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:50.303068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.302958 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q"] Oct 30 00:48:50.318096 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:50.318064 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b84395_88a9_4887_b9ce_5e67e258f1a9.slice/crio-8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68 WatchSource:0}: Error finding container 8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68: Status 404 returned error can't find the container with id 8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68 Oct 30 00:48:50.437725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.437699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:50.729242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.729214 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" event={"ID":"58b84395-88a9-4887-b9ce-5e67e258f1a9","Type":"ContainerStarted","Data":"17370166a32ca472deee5218cfbf933bb87d57f00ba3345956ab5db40f0570ec"} Oct 30 00:48:50.729587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.729272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" event={"ID":"58b84395-88a9-4887-b9ce-5e67e258f1a9","Type":"ContainerStarted","Data":"8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68"} Oct 30 00:48:50.768327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:50.768279 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" podStartSLOduration=1.7682626190000001 podStartE2EDuration="1.768262619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:50.766954984 +0000 UTC m=+983.007190098" watchObservedRunningTime="2026-10-30 00:48:50.768262619 +0000 UTC m=+983.008497737" Oct 30 00:48:51.198669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.198636 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:48:51.203115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.203091 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236176 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236318 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236350 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236390 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236413 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236444 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236513 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236630 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5k7q\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.236768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.236678 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.238732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.238696 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:48:51.337755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337723 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.337906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337759 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.337906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337799 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.337906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337821 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.337906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337852 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.337960 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338114 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5k7q\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338233 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338312 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338632 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338719 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.338802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.338784 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.339899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.339880 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.340190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.340171 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.360373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.360346 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5k7q\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.360497 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.360475 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.528165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.528079 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:51.738300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.738270 5938 generic.go:356] "Generic (PLEG): container finished" podID="58b84395-88a9-4887-b9ce-5e67e258f1a9" containerID="17370166a32ca472deee5218cfbf933bb87d57f00ba3345956ab5db40f0570ec" exitCode=0 Oct 30 00:48:51.738662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.738366 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" event={"ID":"58b84395-88a9-4887-b9ce-5e67e258f1a9","Type":"ContainerDied","Data":"17370166a32ca472deee5218cfbf933bb87d57f00ba3345956ab5db40f0570ec"} Oct 30 00:48:51.844518 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:48:51.844478 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2aeb56_0794_47d6_b61a_ad60e52e47c8.slice/crio-17ca2219ba54d0c0107ad540d63df50f9c58e471f7b83f66948bd29307583627 WatchSource:0}: Error finding container 17ca2219ba54d0c0107ad540d63df50f9c58e471f7b83f66948bd29307583627: Status 404 returned error can't find the container with id 17ca2219ba54d0c0107ad540d63df50f9c58e471f7b83f66948bd29307583627 Oct 30 00:48:51.845243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.845208 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:48:51.847456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.847417 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:51.847530 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.847505 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:51.847577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:51.847552 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:48:52.176076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:52.176011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:52.747725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:52.747684 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" event={"ID":"3d2aeb56-0794-47d6-b61a-ad60e52e47c8","Type":"ContainerStarted","Data":"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78"} Oct 30 00:48:52.748175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:52.747762 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" event={"ID":"3d2aeb56-0794-47d6-b61a-ad60e52e47c8","Type":"ContainerStarted","Data":"17ca2219ba54d0c0107ad540d63df50f9c58e471f7b83f66948bd29307583627"} Oct 30 00:48:52.804888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:52.804199 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podStartSLOduration=1.80415587 podStartE2EDuration="1.80415587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.802597275 +0000 UTC m=+985.042832413" watchObservedRunningTime="2026-10-30 00:48:52.80415587 +0000 UTC m=+985.044390984" Oct 30 00:48:52.977630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:52.977609 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:53.066225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.066140 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\") pod \"58b84395-88a9-4887-b9ce-5e67e258f1a9\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " Oct 30 00:48:53.066366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.066349 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\" (UniqueName: \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\") pod \"58b84395-88a9-4887-b9ce-5e67e258f1a9\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " Oct 30 00:48:53.066583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.066547 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca" pod "58b84395-88a9-4887-b9ce-5e67e258f1a9" (UID: "58b84395-88a9-4887-b9ce-5e67e258f1a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:48:53.066846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.066822 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\") pod \"58b84395-88a9-4887-b9ce-5e67e258f1a9\" (UID: \"58b84395-88a9-4887-b9ce-5e67e258f1a9\") " Oct 30 00:48:53.067630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.067605 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:53.068925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.068899 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m" pod "58b84395-88a9-4887-b9ce-5e67e258f1a9" (UID: "58b84395-88a9-4887-b9ce-5e67e258f1a9"). InnerVolumeSpecName "kube-api-access-ngt5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:48:53.069397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.069376 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca" pod "58b84395-88a9-4887-b9ce-5e67e258f1a9" (UID: "58b84395-88a9-4887-b9ce-5e67e258f1a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:48:53.169297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.169263 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngt5m\" (UniqueName: \"kubernetes.io/projected/58b84395-88a9-4887-b9ce-5e67e258f1a9-kube-api-access-ngt5m\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:53.169297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.169293 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b84395-88a9-4887-b9ce-5e67e258f1a9-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:48:53.529099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.529070 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:53.530790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.530762 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:53.530933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.530812 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:53.757002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.756975 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" Oct 30 00:48:53.757374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.757017 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q" event={"ID":"58b84395-88a9-4887-b9ce-5e67e258f1a9","Type":"ContainerDied","Data":"8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68"} Oct 30 00:48:53.757374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.757068 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d19c7a0566c7b68a911b36fd584c06d8f70eef805c72cc86377d7235acc2a68" Oct 30 00:48:53.816938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:53.816090 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:54.529534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:54.529495 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:54.529707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:54.529569 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:55.528636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:55.528599 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:55.529135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:55.528660 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:55.712150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:55.708514 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:56.529035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:56.528999 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:56.529423 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:56.529053 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:57.278174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:57.278136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:57.529331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:57.529253 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:57.529331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:57.529309 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:58.011090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:58.011056 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:58.150159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:58.150127 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:58.529145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:58.529108 5938 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 00:48:58.529307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:58.529183 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 00:48:58.836582 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:48:58.836474 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:48:58.864194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:58.864166 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:59.533839 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:59.533808 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:59.534310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:59.534290 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:48:59.534970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:48:59.534946 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:49:00.579012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:00.578961 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:02.284588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:02.284559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:03.668561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.668519 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85"] Oct 30 00:49:03.670039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.670013 5938 cpu_manager.go:414] "containerMap: removing container" podUID="58b84395-88a9-4887-b9ce-5e67e258f1a9" containerName="revoke-keys" Oct 30 00:49:03.670039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.670038 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b84395-88a9-4887-b9ce-5e67e258f1a9" containerName="revoke-keys" Oct 30 00:49:03.670377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.670350 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b84395-88a9-4887-b9ce-5e67e258f1a9" containerName="revoke-keys" Oct 30 00:49:03.679062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.679038 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.691579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.691542 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85"] Oct 30 00:49:03.696274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.696252 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:03.696598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.696371 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:03.696686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.696491 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:03.704706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.704684 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:03.831695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.831659 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.831835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.831767 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.831918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.831852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5gw\" (UniqueName: \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.933129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.933055 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5gw\" (UniqueName: \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.933267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.933180 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.933267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.933231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.933818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.933784 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.935670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.935637 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.955538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.955515 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5gw\" (UniqueName: \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\") pod \"maas-api-revoke-sub-ec098887-wcm85\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:03.996219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:03.996198 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:04.023536 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:04.023508 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:04.321149 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:04.321120 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64da4cf_20ca_401d_8861_6e8b8b9341cf.slice/crio-d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576 WatchSource:0}: Error finding container d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576: Status 404 returned error can't find the container with id d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576 Oct 30 00:49:04.321483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:04.321459 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85"] Oct 30 00:49:04.848483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:04.848405 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" event={"ID":"e64da4cf-20ca-401d-8861-6e8b8b9341cf","Type":"ContainerStarted","Data":"23f6fd9670f12fb66a35859824e290d370719444e79aef147c8ab602f366494d"} Oct 30 00:49:04.848483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:04.848462 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" event={"ID":"e64da4cf-20ca-401d-8861-6e8b8b9341cf","Type":"ContainerStarted","Data":"d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576"} Oct 30 00:49:04.888815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:04.888740 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" podStartSLOduration=1.888722366 podStartE2EDuration="1.888722366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.886968023 +0000 UTC m=+997.127203136" watchObservedRunningTime="2026-10-30 00:49:04.888722366 +0000 UTC m=+997.128957480" Oct 30 00:49:05.568208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:05.568120 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:05.946894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:05.945297 5938 generic.go:356] "Generic (PLEG): container finished" podID="e64da4cf-20ca-401d-8861-6e8b8b9341cf" containerID="23f6fd9670f12fb66a35859824e290d370719444e79aef147c8ab602f366494d" exitCode=0 Oct 30 00:49:05.946894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:05.945528 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" event={"ID":"e64da4cf-20ca-401d-8861-6e8b8b9341cf","Type":"ContainerDied","Data":"23f6fd9670f12fb66a35859824e290d370719444e79aef147c8ab602f366494d"} Oct 30 00:49:06.502470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:06.502431 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:06.647887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:06.644527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:07.151391 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.151349 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.214993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.214947 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.231900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.215576 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:49:07.278042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.277951 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:07.294226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.294189 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.298912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.295352 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.551092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.550981 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.580525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.573786 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.629045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.628994 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:07.739029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:07.738967 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:08.088940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.085511 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.217889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.189244 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.281127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.278709 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.281127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.280323 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:49:08.307974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.307529 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.307974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.307789 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:49:08.418886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.418474 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:08.418886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.418548 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:08.418886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.418591 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:08.418886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.418630 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:08.418886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.418668 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:08.459912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.439644 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.623909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.598973 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.623909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.600194 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:08.641324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.625952 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:08.641324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.626025 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:08.685073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.684970 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:08.685401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.685357 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:08.717988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.687155 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:49:08.718438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.718404 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:08.746957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.739721 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:08.746957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.741099 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:08.791899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.769996 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:08.968982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:08.920882 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\" (UniqueName: \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\") pod \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " Oct 30 00:49:09.035984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.021756 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\") pod \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " Oct 30 00:49:09.188805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.168191 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw" pod "e64da4cf-20ca-401d-8861-6e8b8b9341cf" (UID: "e64da4cf-20ca-401d-8861-6e8b8b9341cf"). InnerVolumeSpecName "kube-api-access-fh5gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:09.219964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.207388 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca" pod "e64da4cf-20ca-401d-8861-6e8b8b9341cf" (UID: "e64da4cf-20ca-401d-8861-6e8b8b9341cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:09.281286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.281247 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\") pod \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\" (UID: \"e64da4cf-20ca-401d-8861-6e8b8b9341cf\") " Oct 30 00:49:09.322054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.304037 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca" pod "e64da4cf-20ca-401d-8861-6e8b8b9341cf" (UID: "e64da4cf-20ca-401d-8861-6e8b8b9341cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:09.330422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.330392 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:09.330596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.330583 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e64da4cf-20ca-401d-8861-6e8b8b9341cf-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:09.330675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.330664 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5gw\" (UniqueName: \"kubernetes.io/projected/e64da4cf-20ca-401d-8861-6e8b8b9341cf-kube-api-access-fh5gw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:09.346916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.338407 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.347288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.347255 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:09.374188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.374139 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.401400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.374440 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.427823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.374489 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:49:09.460970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.374514 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.461364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.435026 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.629995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.624989 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:09.629995 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.625053 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.752959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.740825 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.752959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.741226 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.754533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.754487 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:09.813549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.813497 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:09.814129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:09.814095 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:10.339063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.339019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.376286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.376242 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" Oct 30 00:49:10.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.401689 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.563894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.401942 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.568877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.566115 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85" event={"ID":"e64da4cf-20ca-401d-8861-6e8b8b9341cf","Type":"ContainerDied","Data":"d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576"} Oct 30 00:49:10.568877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.566181 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1087f864dcded838197425df5c2b3283691b28a7b855b859bdb71fce890a576" Oct 30 00:49:10.584285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.428407 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.585590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.486982 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.585590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.487019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:10.679937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.676968 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" podUID="671141fa-f7fe-4f3f-952e-bf6aebb07516" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.816004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.815959 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:10.821898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.817536 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:10.845893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.837805 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:49:10.845893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:10.842005 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.141897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.140540 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:11.215263 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.214486 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 00:49:11.356893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.354955 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.683050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.679310 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.683050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.679655 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.683050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.679969 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.687253 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.687223 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:49:11.727729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.715427 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:11.763356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.761520 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:11.804917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.804818 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:11.807819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.807791 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:11.812981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.812857 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:11.849004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.846821 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:11.906887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.891318 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:11.906887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.891401 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:11.906887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.891434 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:11.906887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.891461 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:11.906887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:11.892735 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:12.305735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.305704 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:12.701046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.701017 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:49:12.701635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.701621 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e64da4cf-20ca-401d-8861-6e8b8b9341cf" containerName="revoke-keys" Oct 30 00:49:12.701697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.701635 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64da4cf-20ca-401d-8861-6e8b8b9341cf" containerName="revoke-keys" Oct 30 00:49:12.701772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.701763 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64da4cf-20ca-401d-8861-6e8b8b9341cf" containerName="revoke-keys" Oct 30 00:49:12.706213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.706189 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.734007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.733978 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:49:12.736929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.736882 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ba85458-openshift-default-dockercfg-gzjhg\"" Oct 30 00:49:12.794220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.794190 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.830683 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.830733 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w77r\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.830782 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.830877 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.830909 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.831002 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.831083 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.831193 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.834893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.831310 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.937888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.937828 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.938065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.937898 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.938065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938010 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.938207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938083 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.938207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938177 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938477 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w77r\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.938537 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.939222 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.958947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.939479 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.995266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.965220 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.995266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.965545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.995266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.966040 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.995266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.990757 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:12.995514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:12.995371 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:13.014525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.014342 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:13.017732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.017702 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:13.018646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.018616 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w77r\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\") pod \"e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:13.022691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.022616 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:13.223830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.223746 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:13.387676 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:13.387648 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca72d5aa_2556_4ddc_beab_0b8225de79ec.slice/crio-f1db445840f1db91fde052a4934b0b8ee10a2eca5fb3bff4c1c93e749809ebba WatchSource:0}: Error finding container f1db445840f1db91fde052a4934b0b8ee10a2eca5fb3bff4c1c93e749809ebba: Status 404 returned error can't find the container with id f1db445840f1db91fde052a4934b0b8ee10a2eca5fb3bff4c1c93e749809ebba Oct 30 00:49:13.388709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.388618 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:49:13.390642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.390606 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:13.390734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.390671 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:13.390734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.390702 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:13.806340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.806299 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerStarted","Data":"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6"} Oct 30 00:49:13.806721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.806391 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerStarted","Data":"f1db445840f1db91fde052a4934b0b8ee10a2eca5fb3bff4c1c93e749809ebba"} Oct 30 00:49:13.864545 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:13.864500 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" podStartSLOduration=1.864487289 podStartE2EDuration="1.864487289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:13.861052756 +0000 UTC m=+1006.101287883" watchObservedRunningTime="2026-10-30 00:49:13.864487289 +0000 UTC m=+1006.104722403" Oct 30 00:49:14.029442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.029404 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:14.038910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.037722 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:14.147998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.147963 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46"] Oct 30 00:49:14.152700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.152678 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.167318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.167294 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:14.167677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.167656 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:14.167677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.167658 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:14.167832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.167702 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:14.174149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.174124 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46"] Oct 30 00:49:14.207489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.207453 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.262249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.262226 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.262352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.262258 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpcp\" (UniqueName: \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.262395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.262371 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.349986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.349954 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.364288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.364258 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.364384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.364304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpcp\" (UniqueName: \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.364384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.364378 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.364811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.364789 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.366582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.366560 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.386618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.386589 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpcp\" (UniqueName: \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\") pod \"maas-api-revoke-sub-1dbe2ff0-nlc46\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.467755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.467691 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:14.542282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.542248 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.693162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.693127 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.775370 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:14.775337 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7acdeb8_6605_4304_bc5b_40f4125e9114.slice/crio-37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd WatchSource:0}: Error finding container 37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd: Status 404 returned error can't find the container with id 37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd Oct 30 00:49:14.775506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.775413 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46"] Oct 30 00:49:14.818276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.818246 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" event={"ID":"a7acdeb8-6605-4304-bc5b-40f4125e9114","Type":"ContainerStarted","Data":"37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd"} Oct 30 00:49:14.818714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.818691 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:14.819540 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.819522 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:49:14.851225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.851205 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.990721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:14.990686 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.207209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.207186 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.262140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.262108 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:49:15.267812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.267786 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.284976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.284948 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cab9990d-gw-openshift-default-dockercfg-7h89s\"" Oct 30 00:49:15.294495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.294009 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:49:15.382887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.372146 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423550 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423643 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423698 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423756 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423815 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423919 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.423972 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.424046 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.429096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.424157 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzdm\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533006 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533073 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533140 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533237 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzdm\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533405 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533489 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533539 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533585 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.533640 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.534140 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.534936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.534381 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.549884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.542548 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.549884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.544592 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.549884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.544852 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.549884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.545108 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.549884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.545335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.571755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.569320 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.572315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.572283 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzdm\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.605015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.604986 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:15.757623 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.757586 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.828303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.828272 5938 generic.go:356] "Generic (PLEG): container finished" podID="a7acdeb8-6605-4304-bc5b-40f4125e9114" containerID="f12bcd1b926d40aa916483f74b50c9a755d790d98a32523bc61d8b3d9ed2d492" exitCode=0 Oct 30 00:49:15.828696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.828360 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" event={"ID":"a7acdeb8-6605-4304-bc5b-40f4125e9114","Type":"ContainerDied","Data":"f12bcd1b926d40aa916483f74b50c9a755d790d98a32523bc61d8b3d9ed2d492"} Oct 30 00:49:15.955393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:15.955315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:16.016430 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:16.004033 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84070897_5bc1_426b_ba2f_884efde7bc03.slice/crio-6af812213f654f68f0e6b66615b15c891f1e5b2641010dc798e63d096682546e WatchSource:0}: Error finding container 6af812213f654f68f0e6b66615b15c891f1e5b2641010dc798e63d096682546e: Status 404 returned error can't find the container with id 6af812213f654f68f0e6b66615b15c891f1e5b2641010dc798e63d096682546e Oct 30 00:49:16.024999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.018849 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:49:16.043091 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.043039 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:16.044927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.044842 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:16.045100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.045079 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:16.837791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.837753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerStarted","Data":"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef"} Oct 30 00:49:16.838321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.837809 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerStarted","Data":"6af812213f654f68f0e6b66615b15c891f1e5b2641010dc798e63d096682546e"} Oct 30 00:49:16.955916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.955840 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podStartSLOduration=1.955821231 podStartE2EDuration="1.955821231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:16.936624331 +0000 UTC m=+1009.176859466" watchObservedRunningTime="2026-10-30 00:49:16.955821231 +0000 UTC m=+1009.196056345" Oct 30 00:49:16.998734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:16.998711 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:17.168192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.168155 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\") pod \"a7acdeb8-6605-4304-bc5b-40f4125e9114\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " Oct 30 00:49:17.168410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.168220 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\") pod \"a7acdeb8-6605-4304-bc5b-40f4125e9114\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " Oct 30 00:49:17.168410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.168267 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\" (UniqueName: \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\") pod \"a7acdeb8-6605-4304-bc5b-40f4125e9114\" (UID: \"a7acdeb8-6605-4304-bc5b-40f4125e9114\") " Oct 30 00:49:17.168631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.168609 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca" pod "a7acdeb8-6605-4304-bc5b-40f4125e9114" (UID: "a7acdeb8-6605-4304-bc5b-40f4125e9114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:17.169176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.169147 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:17.170585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.170562 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp" pod "a7acdeb8-6605-4304-bc5b-40f4125e9114" (UID: "a7acdeb8-6605-4304-bc5b-40f4125e9114"). InnerVolumeSpecName "kube-api-access-kqpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:17.170729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.170711 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca" pod "a7acdeb8-6605-4304-bc5b-40f4125e9114" (UID: "a7acdeb8-6605-4304-bc5b-40f4125e9114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:17.271061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.271031 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7acdeb8-6605-4304-bc5b-40f4125e9114-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:17.271061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.271061 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpcp\" (UniqueName: \"kubernetes.io/projected/a7acdeb8-6605-4304-bc5b-40f4125e9114-kube-api-access-kqpcp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:17.303837 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.303805 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.605803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.605774 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:17.612144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.612118 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:17.717875 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.717827 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.849003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.848969 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" Oct 30 00:49:17.849003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.848989 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46" event={"ID":"a7acdeb8-6605-4304-bc5b-40f4125e9114","Type":"ContainerDied","Data":"37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd"} Oct 30 00:49:17.849523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.849105 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ee975c71ecaaba88bafb533f1afde1c3c060e485254634618a176ffff184fd" Oct 30 00:49:17.849586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.849562 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:17.851045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.851024 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:49:17.881472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:17.881450 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:18.898741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:18.898705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:19.039473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:19.039445 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:19.179673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:19.179608 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:20.417267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:20.417235 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:20.560342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:20.560307 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:21.542969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.542935 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:21.571198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.571158 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl"] Oct 30 00:49:21.572444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.572295 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a7acdeb8-6605-4304-bc5b-40f4125e9114" containerName="revoke-keys" Oct 30 00:49:21.572444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.572324 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7acdeb8-6605-4304-bc5b-40f4125e9114" containerName="revoke-keys" Oct 30 00:49:21.572917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.572499 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7acdeb8-6605-4304-bc5b-40f4125e9114" containerName="revoke-keys" Oct 30 00:49:21.578555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.578527 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.600177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.600148 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:21.600456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.600435 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:21.600652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.600633 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:21.608976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.606925 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl"] Oct 30 00:49:21.608976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.607903 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:21.617664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.617422 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.617664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.617497 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.617854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.617792 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppddx\" (UniqueName: \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.719490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.719459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.719674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.719511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.719719 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.719686 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppddx\" (UniqueName: \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.720307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.720284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.722003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.721983 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.742440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.742407 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppddx\" (UniqueName: \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\") pod \"maas-api-revoke-sub-7a9e216c-vk7dl\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:21.906943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:21.906918 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:22.216395 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:22.216365 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de WatchSource:0}: Error finding container 45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de: Status 404 returned error can't find the container with id 45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de Oct 30 00:49:22.218368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:22.218254 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl"] Oct 30 00:49:22.907566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:22.907516 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" event={"ID":"aaee8520-fb23-4258-9ff2-a2e1510eb145","Type":"ContainerStarted","Data":"21f74b6080c28e2ff4584a5f07e35372cd5104ea2616725e4110e53f8a64bcfc"} Oct 30 00:49:22.907566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:22.907569 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" event={"ID":"aaee8520-fb23-4258-9ff2-a2e1510eb145","Type":"ContainerStarted","Data":"45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de"} Oct 30 00:49:22.950274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:22.950212 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" podStartSLOduration=1.950192104 podStartE2EDuration="1.950192104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:22.946709512 +0000 UTC m=+1015.186944627" watchObservedRunningTime="2026-10-30 00:49:22.950192104 +0000 UTC m=+1015.190427218" Oct 30 00:49:23.278139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:23.278043 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:24.025887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:24.021823 5938 generic.go:356] "Generic (PLEG): container finished" podID="aaee8520-fb23-4258-9ff2-a2e1510eb145" containerID="21f74b6080c28e2ff4584a5f07e35372cd5104ea2616725e4110e53f8a64bcfc" exitCode=0 Oct 30 00:49:24.025887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:24.021880 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" event={"ID":"aaee8520-fb23-4258-9ff2-a2e1510eb145","Type":"ContainerDied","Data":"21f74b6080c28e2ff4584a5f07e35372cd5104ea2616725e4110e53f8a64bcfc"} Oct 30 00:49:25.699922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.695951 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 00:49:25.699922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.696062 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:25.699922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.696453 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:49:25.699922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.697129 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:25.788702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.785975 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:25.788702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.786379 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:49:25.825885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.809436 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:25.825885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.824175 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:25.897830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.895953 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:25.897830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.897009 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:25.986942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:25.985921 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:26.369076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.367933 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:26.482750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.482716 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:26.487855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.487808 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:26.767888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.746119 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\" (UniqueName: \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\") pod \"aaee8520-fb23-4258-9ff2-a2e1510eb145\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " Oct 30 00:49:27.309888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.288777 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\") pod \"aaee8520-fb23-4258-9ff2-a2e1510eb145\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " Oct 30 00:49:27.360772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.358589 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\") pod \"aaee8520-fb23-4258-9ff2-a2e1510eb145\" (UID: \"aaee8520-fb23-4258-9ff2-a2e1510eb145\") " Oct 30 00:49:27.390026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.769045 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:49:27.390026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.387322 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.438891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.768813 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:27.494921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.790239 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:27.605885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.821677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:27.605885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.822401 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.836299 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.901413 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.901450 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.779412 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx" pod "aaee8520-fb23-4258-9ff2-a2e1510eb145" (UID: "aaee8520-fb23-4258-9ff2-a2e1510eb145"). InnerVolumeSpecName "kube-api-access-ppddx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.308035 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.612350 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.619905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.768562 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:26.790200 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.440170 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621481 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621562 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621634 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621690 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621744 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:27.634822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.621794 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:27.638012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.636279 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:27.638012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.636351 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:27.638012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.636405 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:27.638012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.463079 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca" pod "aaee8520-fb23-4258-9ff2-a2e1510eb145" (UID: "aaee8520-fb23-4258-9ff2-a2e1510eb145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:27.638012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.547517 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca" pod "aaee8520-fb23-4258-9ff2-a2e1510eb145" (UID: "aaee8520-fb23-4258-9ff2-a2e1510eb145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:27.761102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.759708 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" event={"ID":"aaee8520-fb23-4258-9ff2-a2e1510eb145","Type":"ContainerDied","Data":"45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de"} Oct 30 00:49:27.761102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.759778 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de" Oct 30 00:49:27.761102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.759889 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl" Oct 30 00:49:27.782221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.782190 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:27.782727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.782707 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppddx\" (UniqueName: \"kubernetes.io/projected/aaee8520-fb23-4258-9ff2-a2e1510eb145-kube-api-access-ppddx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:27.782843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.782828 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaee8520-fb23-4258-9ff2-a2e1510eb145-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:27.941824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:27.940559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:28.028473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.028448 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:28.134890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.134818 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:28.144855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.140681 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:28.144855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.140734 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:28.144855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.141446 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:28.144855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.141491 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:28.152650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.152555 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:28.153247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.152935 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:28.153514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.153439 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:28.154101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.153679 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:28.470696 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:28.470661 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:28.486917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:28.486882 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:28.838236 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:28.838147 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:49:30.049681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:30.049613 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:30.084470 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:30.084431 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:31.645990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.645950 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:31.785140 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.785108 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:31.931065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.930959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:31.974265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.974240 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt"] Oct 30 00:49:31.974908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.974894 5938 cpu_manager.go:414] "containerMap: removing container" podUID="aaee8520-fb23-4258-9ff2-a2e1510eb145" containerName="revoke-keys" Oct 30 00:49:31.974951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.974909 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaee8520-fb23-4258-9ff2-a2e1510eb145" containerName="revoke-keys" Oct 30 00:49:31.975026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.975018 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaee8520-fb23-4258-9ff2-a2e1510eb145" containerName="revoke-keys" Oct 30 00:49:31.980321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.980300 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:31.998317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:31.998293 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt"] Oct 30 00:49:32.003467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.003444 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:32.003597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.003518 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:32.003597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.003537 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:32.005917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.005898 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:32.074184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.074162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxl9v\" (UniqueName: \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.074381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.074358 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.074524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.074505 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.175856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.175832 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxl9v\" (UniqueName: \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.175990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.175979 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.176082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.176065 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.176843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.176816 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.178382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.178363 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.198222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.198172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxl9v\" (UniqueName: \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\") pod \"maas-api-revoke-sub-c6b4ba46-sgmzt\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.294060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.294038 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:32.612382 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:32.612351 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c79c72e_900b_4d41_9fc7_ef8c82e9fee3.slice/crio-3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594 WatchSource:0}: Error finding container 3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594: Status 404 returned error can't find the container with id 3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594 Oct 30 00:49:32.612484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.612388 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt"] Oct 30 00:49:32.809962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:32.809922 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" event={"ID":"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3","Type":"ContainerStarted","Data":"3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594"} Oct 30 00:49:33.366475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:33.365650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:33.819444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:33.819367 5938 generic.go:356] "Generic (PLEG): container finished" podID="5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" containerID="21e23dce8ad8cf74929aba9b9bbd6c2e6bf3c5c90a6d7dcd5aa9c64bf4b70430" exitCode=0 Oct 30 00:49:33.819803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:33.819445 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" event={"ID":"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3","Type":"ContainerDied","Data":"21e23dce8ad8cf74929aba9b9bbd6c2e6bf3c5c90a6d7dcd5aa9c64bf4b70430"} Oct 30 00:49:34.885839 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:34.885720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:34.992516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:34.992480 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 00:49:34.999217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:34.999170 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.018714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.018685 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 00:49:35.021214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.021189 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"distinct-2-w3-cefb27-kserve-self-signed-certs\"" Oct 30 00:49:35.025359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025333 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfd8\" (UniqueName: \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.025513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.025570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025532 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.025671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025656 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.025832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025800 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.025965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.025913 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.036613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.036594 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:35.072543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.072516 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:35.127549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.127514 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\") pod \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " Oct 30 00:49:35.127716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.127696 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\" (UniqueName: \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\") pod \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " Oct 30 00:49:35.127809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.127773 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\") pod \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\" (UID: \"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3\") " Oct 30 00:49:35.127884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.127826 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca" pod "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" (UID: "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:35.128048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128026 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.128181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128162 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.128441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128420 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.128530 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128502 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.128606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128585 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.128922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.128900 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.129175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.129152 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfd8\" (UniqueName: \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.129272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.129246 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.129331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.129274 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.129417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.129401 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:35.130244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.130216 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca" pod "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" (UID: "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:35.130529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.130508 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v" pod "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" (UID: "5c79c72e-900b-4d41-9fc7-ef8c82e9fee3"). InnerVolumeSpecName "kube-api-access-lxl9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:35.131884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.131846 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.132030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.132007 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.151838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.151815 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfd8\" (UniqueName: \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\") pod \"distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.255257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.237540 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxl9v\" (UniqueName: \"kubernetes.io/projected/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-kube-api-access-lxl9v\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:35.255257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.237577 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:35.335357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.335325 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:35.644899 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:35.644849 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2160acac_590a_4dd3_bb22_23e6f85ffcb7.slice/crio-e8291e64593a0ddc9d830bf16a4ea8f0f14318b8d125b740839f995696e5e49d WatchSource:0}: Error finding container e8291e64593a0ddc9d830bf16a4ea8f0f14318b8d125b740839f995696e5e49d: Status 404 returned error can't find the container with id e8291e64593a0ddc9d830bf16a4ea8f0f14318b8d125b740839f995696e5e49d Oct 30 00:49:35.646848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.646827 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 00:49:35.846277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.846242 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" event={"ID":"2160acac-590a-4dd3-bb22-23e6f85ffcb7","Type":"ContainerStarted","Data":"7be5ac414b094dad1b4d200282fcc225889754eaf6653cc88d8c9186b92e9468"} Oct 30 00:49:35.846448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.846293 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" event={"ID":"2160acac-590a-4dd3-bb22-23e6f85ffcb7","Type":"ContainerStarted","Data":"e8291e64593a0ddc9d830bf16a4ea8f0f14318b8d125b740839f995696e5e49d"} Oct 30 00:49:35.847908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.847885 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" Oct 30 00:49:35.847908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.847892 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt" event={"ID":"5c79c72e-900b-4d41-9fc7-ef8c82e9fee3","Type":"ContainerDied","Data":"3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594"} Oct 30 00:49:35.848072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.847935 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ecdd0dc87fdc9369d0bd89fe1d240751fcf5a5cc3316a13c1a184176bc32594" Oct 30 00:49:35.886200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:35.886151 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podStartSLOduration=1.8861341299999999 podStartE2EDuration="1.88613413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:35.885702796 +0000 UTC m=+1028.125937922" watchObservedRunningTime="2026-10-30 00:49:35.88613413 +0000 UTC m=+1028.126369243" Oct 30 00:49:36.043344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.043243 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn"] Oct 30 00:49:36.050453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.049517 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e3f943-tdtbn"] Oct 30 00:49:36.066561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.066533 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:36.335513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.335442 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:36.646543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.646484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:36.898054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:36.897960 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:37.882358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:37.882324 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae" path="/var/lib/kubelet/pods/0b7bca6c-c5f6-4ce7-99ee-83920d1fdfae/volumes" Oct 30 00:49:38.396904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:38.396846 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:38.570676 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:38.570637 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:40.066051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:40.066016 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:41.668827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:41.668791 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:43.387019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:43.386984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:44.254038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:44.253979 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:44.405038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:44.404972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:44.547760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:44.547680 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:45.068564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:45.068529 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:45.184407 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:45.184359 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:45.335813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:45.335727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:45.348121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:45.348100 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:45.952176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:45.952148 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:46.314715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.314634 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc"] Oct 30 00:49:46.315381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.315360 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" containerName="revoke-keys" Oct 30 00:49:46.315381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.315382 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" containerName="revoke-keys" Oct 30 00:49:46.315567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.315552 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" containerName="revoke-keys" Oct 30 00:49:46.319358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.319337 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.339437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.339409 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc"] Oct 30 00:49:46.344049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.344028 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:46.344049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.344028 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:46.344220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.344037 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:46.344276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.344218 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:46.403099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.403070 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.403213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.403105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5j2j\" (UniqueName: \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.403289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.403271 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.504404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.504379 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.504520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.504412 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5j2j\" (UniqueName: \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.504520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.504504 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.505158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.505140 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.506790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.506762 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.526802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.526777 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5j2j\" (UniqueName: \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\") pod \"maas-api-revoke-sub-c88289bd-zhrsc\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.634212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.634195 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:46.937438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.937410 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc"] Oct 30 00:49:46.937941 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:46.937918 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef439a8_55e3_417b_b344_4122ce0c81b9.slice/crio-1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8 WatchSource:0}: Error finding container 1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8: Status 404 returned error can't find the container with id 1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8 Oct 30 00:49:46.953844 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.953807 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" event={"ID":"3ef439a8-55e3-417b-b344-4122ce0c81b9","Type":"ContainerStarted","Data":"1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8"} Oct 30 00:49:46.974366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:46.973275 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:47.963588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:47.963557 5938 generic.go:356] "Generic (PLEG): container finished" podID="3ef439a8-55e3-417b-b344-4122ce0c81b9" containerID="3ed6cc0bdb18622edb40369b10bab1e390bd9fdc19d26d53771fc56cdf2103c0" exitCode=0 Oct 30 00:49:47.964059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:47.963631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" event={"ID":"3ef439a8-55e3-417b-b344-4122ce0c81b9","Type":"ContainerDied","Data":"3ed6cc0bdb18622edb40369b10bab1e390bd9fdc19d26d53771fc56cdf2103c0"} Oct 30 00:49:48.561112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:48.561046 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:48.588671 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:48.588637 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:49.089557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.089514 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n"] Oct 30 00:49:49.146925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.129717 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c821316-x7j2n"] Oct 30 00:49:49.352357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.352333 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:49.466274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.466180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\") pod \"3ef439a8-55e3-417b-b344-4122ce0c81b9\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " Oct 30 00:49:49.466432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.466335 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\") pod \"3ef439a8-55e3-417b-b344-4122ce0c81b9\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " Oct 30 00:49:49.466432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.466375 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\" (UniqueName: \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\") pod \"3ef439a8-55e3-417b-b344-4122ce0c81b9\" (UID: \"3ef439a8-55e3-417b-b344-4122ce0c81b9\") " Oct 30 00:49:49.466785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.466740 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca" pod "3ef439a8-55e3-417b-b344-4122ce0c81b9" (UID: "3ef439a8-55e3-417b-b344-4122ce0c81b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:49:49.467164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.467140 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:49.468309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.468281 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca" pod "3ef439a8-55e3-417b-b344-4122ce0c81b9" (UID: "3ef439a8-55e3-417b-b344-4122ce0c81b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:49:49.468437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.468420 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j" pod "3ef439a8-55e3-417b-b344-4122ce0c81b9" (UID: "3ef439a8-55e3-417b-b344-4122ce0c81b9"). InnerVolumeSpecName "kube-api-access-k5j2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:49:49.569061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.569020 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5j2j\" (UniqueName: \"kubernetes.io/projected/3ef439a8-55e3-417b-b344-4122ce0c81b9-kube-api-access-k5j2j\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:49.569061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.569059 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ef439a8-55e3-417b-b344-4122ce0c81b9-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:49:49.881144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:49.881099 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2fbfcc-5218-4d5a-a512-161782d706a3" path="/var/lib/kubelet/pods/2b2fbfcc-5218-4d5a-a512-161782d706a3/volumes" Oct 30 00:49:50.035888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.035410 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" event={"ID":"3ef439a8-55e3-417b-b344-4122ce0c81b9","Type":"ContainerDied","Data":"1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8"} Oct 30 00:49:50.035888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.035472 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c9393cc29497f6a56f23445c808e90a9cc4189f1c8a560ff9b94ad7e7c886f8" Oct 30 00:49:50.035888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.035541 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc" Oct 30 00:49:50.770566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.770532 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:49:50.772272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.772246 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3ef439a8-55e3-417b-b344-4122ce0c81b9" containerName="revoke-keys" Oct 30 00:49:50.772425 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.772411 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef439a8-55e3-417b-b344-4122ce0c81b9" containerName="revoke-keys" Oct 30 00:49:50.772705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.772691 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef439a8-55e3-417b-b344-4122ce0c81b9" containerName="revoke-keys" Oct 30 00:49:50.832406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.821271 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:49:50.832406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.821398 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.891095 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.892916 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.892956 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.892987 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893025 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893117 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893197 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893294 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmt7\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893371 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:50.909801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:50.893510 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012043 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012188 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012219 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012262 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012311 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012537 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmt7\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.012627 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.014921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.014388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.029946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.026908 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.029946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.027474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.029946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.027709 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.048652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.048344 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.235961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.235042 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.321066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.320950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmt7\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.371908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.371856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.527888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.502454 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:51.583315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:51.583212 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:52.137890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.109125 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.164088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.164039 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.192643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.192602 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.234888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.232979 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:52.256989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.256944 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.257591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.257560 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.287443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.287401 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:49:52.288026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.287997 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:49:52.288656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.288630 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:52.303809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.291909 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:52.304911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.291933 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:52.914890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:52.889284 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.188533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.076312 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.188533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.144490 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:49:53.188533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.165966 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:53.188533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.167357 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.188533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.167613 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:53.208939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.193549 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.223345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.213098 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:53.269153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.202946 5938 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:53.269153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.268113 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.269153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.263355 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.269153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.252038 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:49:53.269811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.263391 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:49:53.270773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.270755 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:53.272201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.272188 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:53.301363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.301342 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:53.302167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.301077 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.302534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.301113 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:49:53.321501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.321473 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:53.321743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.321727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:53.354985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.354930 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:53.364006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.363949 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:49:53.364574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.364537 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:53.388920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.368507 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:53.388920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.368575 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:53.388920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.368610 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:53.412213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.412170 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:53.412530 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.412502 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:53.412726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.412693 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:49:53.752927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.729595 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:49:53.871907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.862958 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:53.943894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:53.943826 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:54.024909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.018264 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 00:49:54.024909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.018518 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 00:49:54.077471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.077207 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:49:54.077471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.077308 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.221159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.161177 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.221159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.173213 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.247153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.246537 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.269769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.269016 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.280428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.270054 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:49:54.280428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.270533 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.460619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.385075 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:54.460619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.386430 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:54.460619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.386682 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:49:54.460619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.402339 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:49:54.460619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.402565 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:49:54.593890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.569239 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:54.709535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.627653 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" event={"ID":"404212e5-c892-4aad-9dc0-49cdcd1bed75","Type":"ContainerStarted","Data":"c971d3c374ffbad9731927e78559c1adefd1d81f3cea39ca2eb493604a5e347a"} Oct 30 00:49:54.711302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:54.627970 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:55.223718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.138200 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xlz62" podUID="671141fa-f7fe-4f3f-952e-bf6aebb07516" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.19:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:55.406936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.405985 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:55.406936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.406350 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:49:55.406936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.406687 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:55.407893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.407540 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:49:55.407893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.407801 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:49:55.410394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.408287 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:49:55.410394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.408548 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:49:55.412828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.410639 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:49:55.412828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.410932 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:49:55.433571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.429479 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:49:55.433571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.429769 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:49:55.701888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.696433 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:49:55.785890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.764161 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:49:55.852737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.852335 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:49:55.918896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:55.893890 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:56.022030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.001346 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 00:49:56.172931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.172903 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z"] Oct 30 00:49:56.173162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173128 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae8f8554-9s74z"] Oct 30 00:49:56.173302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173162 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" event={"ID":"404212e5-c892-4aad-9dc0-49cdcd1bed75","Type":"ContainerStarted","Data":"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8"} Oct 30 00:49:56.173302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173241 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:49:56.173302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173285 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:49:56.173302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173316 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:49:56.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173350 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:49:56.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173383 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 00:49:56.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173416 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:49:56.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173447 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:49:56.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.173478 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:49:56.268529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.268480 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podStartSLOduration=6.268466989 podStartE2EDuration="6.268466989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:56.222487332 +0000 UTC m=+1048.462722447" watchObservedRunningTime="2026-10-30 00:49:56.268466989 +0000 UTC m=+1048.508702166" Oct 30 00:49:56.669114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.669083 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:49:56.672513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.672385 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:49:56.672513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.672445 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:49:56.881577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:56.881528 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:57.083405 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.083298 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg"] Oct 30 00:49:57.244889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.220296 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.265797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.264455 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:49:57.265797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.265194 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:49:57.265797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.265274 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:49:57.265797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.265339 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:49:57.282044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.272265 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg"] Oct 30 00:49:57.377587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.377553 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.378286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.378256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dnn\" (UniqueName: \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.379238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.379215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.508888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.508450 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.508888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.508578 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.508888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.508816 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dnn\" (UniqueName: \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.514024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.513996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.544595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.544565 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.549807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.549752 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dnn\" (UniqueName: \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\") pod \"maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.621888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.609971 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:49:57.621888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.610030 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:49:57.688274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.688178 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:49:57.869901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.862303 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:57.976625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:57.976545 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c3fa73-9355-4821-b0b3-7ddfe2970beb" path="/var/lib/kubelet/pods/96c3fa73-9355-4821-b0b3-7ddfe2970beb/volumes" Oct 30 00:49:58.642888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:58.623929 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:49:58.642888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:58.623982 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:49:58.661889 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:49:58.660656 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7606b5_8833_494a_9670_24e1253bc8a3.slice/crio-46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc WatchSource:0}: Error finding container 46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc: Status 404 returned error can't find the container with id 46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc Oct 30 00:49:58.682163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:58.681999 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg"] Oct 30 00:49:58.973214 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:58.973060 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:49:59.076912 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:49:59.075914 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:49:59.363515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.363473 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" event={"ID":"ad7606b5-8833-494a-9670-24e1253bc8a3","Type":"ContainerStarted","Data":"79028b273516a60422cd13f15a3d4c2ece3c50231ab7ea98781d0b489c93db39"} Oct 30 00:49:59.363804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.363778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" event={"ID":"ad7606b5-8833-494a-9670-24e1253bc8a3","Type":"ContainerStarted","Data":"46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc"} Oct 30 00:49:59.431209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.431151 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" podStartSLOduration=3.431130494 podStartE2EDuration="3.431130494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:59.424175966 +0000 UTC m=+1051.664411080" watchObservedRunningTime="2026-10-30 00:49:59.431130494 +0000 UTC m=+1051.671365608" Oct 30 00:49:59.653614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.652493 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:59.654369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.654330 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:49:59.654604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.654567 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:49:59.904310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:49:59.898530 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:00.114283 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:00.112348 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:00.374674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.374642 5938 generic.go:356] "Generic (PLEG): container finished" podID="ad7606b5-8833-494a-9670-24e1253bc8a3" containerID="79028b273516a60422cd13f15a3d4c2ece3c50231ab7ea98781d0b489c93db39" exitCode=0 Oct 30 00:50:00.374843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.374707 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" event={"ID":"ad7606b5-8833-494a-9670-24e1253bc8a3","Type":"ContainerDied","Data":"79028b273516a60422cd13f15a3d4c2ece3c50231ab7ea98781d0b489c93db39"} Oct 30 00:50:00.436846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.436810 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:00.449037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.449010 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.463826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.463796 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:00.538957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.538917 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539120 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539176 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539304 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539426 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539464 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjlt7\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539618 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539672 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.539794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.539775 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.603903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.598199 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:50:00.603903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.598254 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:50:00.640991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.640910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.640991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.640960 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641006 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641109 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjlt7\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641222 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641314 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641387 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641569 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641703 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.641758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641748 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.642068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.641811 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.643804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.643781 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.644160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.644140 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.663585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.663557 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjlt7\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.664271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.664249 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.832652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.832616 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:00.861026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.861003 5938 patch_prober.go:29] interesting pod/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:50:00.861127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:00.861053 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:01.031494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.031459 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7"] Oct 30 00:50:01.036610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.036586 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bedad60-drxs7"] Oct 30 00:50:01.186885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.185930 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:01.186885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.186090 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:01.186885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.186193 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:01.216890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.216557 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:01.423048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.422958 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" event={"ID":"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc","Type":"ContainerStarted","Data":"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72"} Oct 30 00:50:01.423048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.423013 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" event={"ID":"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc","Type":"ContainerStarted","Data":"f052f95267816efe5794268f70ac54d0aee539b02645f71f34fc118ac7b0d960"} Oct 30 00:50:01.515294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.514236 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podStartSLOduration=1.51421392 podStartE2EDuration="1.51421392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:01.481201489 +0000 UTC m=+1053.721436604" watchObservedRunningTime="2026-10-30 00:50:01.51421392 +0000 UTC m=+1053.754449033" Oct 30 00:50:01.608885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.588233 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:50:01.608885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.588293 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:50:01.675763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.660118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:01.757799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.757451 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:50:01.833041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.832943 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:01.865490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.865450 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\") pod \"ad7606b5-8833-494a-9670-24e1253bc8a3\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " Oct 30 00:50:01.865672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.865513 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\") pod \"ad7606b5-8833-494a-9670-24e1253bc8a3\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " Oct 30 00:50:01.865672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.865663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\" (UniqueName: \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\") pod \"ad7606b5-8833-494a-9670-24e1253bc8a3\" (UID: \"ad7606b5-8833-494a-9670-24e1253bc8a3\") " Oct 30 00:50:01.868185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.868153 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca" pod "ad7606b5-8833-494a-9670-24e1253bc8a3" (UID: "ad7606b5-8833-494a-9670-24e1253bc8a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:50:01.883764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.883607 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca" pod "ad7606b5-8833-494a-9670-24e1253bc8a3" (UID: "ad7606b5-8833-494a-9670-24e1253bc8a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:50:01.883764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.883711 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn" pod "ad7606b5-8833-494a-9670-24e1253bc8a3" (UID: "ad7606b5-8833-494a-9670-24e1253bc8a3"). InnerVolumeSpecName "kube-api-access-q7dnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:01.925003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.924167 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c57169-dc49-44f5-bceb-8de46ecaf21c" path="/var/lib/kubelet/pods/b8c57169-dc49-44f5-bceb-8de46ecaf21c/volumes" Oct 30 00:50:01.969522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.969491 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dnn\" (UniqueName: \"kubernetes.io/projected/ad7606b5-8833-494a-9670-24e1253bc8a3-kube-api-access-q7dnn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:01.969522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.969521 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:01.969716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:01.969533 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7606b5-8833-494a-9670-24e1253bc8a3-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:02.455882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.451037 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" Oct 30 00:50:02.455882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.451634 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg" event={"ID":"ad7606b5-8833-494a-9670-24e1253bc8a3","Type":"ContainerDied","Data":"46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc"} Oct 30 00:50:02.455882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.451675 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46993663f3342385e42524b9edc8ffbcacc046349e0b9335e96f0d47a74b96fc" Oct 30 00:50:02.589725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.589302 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 00:50:02.589725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.589360 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 00:50:02.834812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.834723 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 00:50:02.834812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:02.834785 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 00:50:03.116888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.116317 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:03.502501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.502391 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:03.608890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.602751 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:50:03.608890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.607575 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:50:03.615531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.615445 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:50:03.833827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.833744 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 00:50:03.834241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.834211 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 00:50:03.897265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.896973 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:50:03.897265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:03.897024 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:04.833699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:04.833663 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 00:50:04.833937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:04.833756 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 00:50:05.211898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:05.209946 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:05.833708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:05.833672 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 00:50:05.833921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:05.833748 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 00:50:06.833389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:06.833353 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 00:50:06.841958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:06.833425 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 00:50:07.205381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.205303 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:07.527736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.527652 5938 generic.go:356] "Generic (PLEG): container finished" podID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerID="2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6" exitCode=0 Oct 30 00:50:07.527911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.527738 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerDied","Data":"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6"} Oct 30 00:50:07.528221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.528201 5938 scope.go:122] "RemoveContainer" containerID="2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6" Oct 30 00:50:07.529813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.529794 5938 generic.go:356] "Generic (PLEG): container finished" podID="84070897-5bc1-426b-ba2f-884efde7bc03" containerID="6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef" exitCode=0 Oct 30 00:50:07.529956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.529828 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.529956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.529845 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerDied","Data":"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef"} Oct 30 00:50:07.529956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.529934 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.530131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.529975 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.530268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.530251 5938 scope.go:122] "RemoveContainer" containerID="6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef" Oct 30 00:50:07.531824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.531797 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.531930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.531912 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.531983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.531969 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:07.683647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.683599 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:07.840953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.829413 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:07.913367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.911120 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:07.913367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.911999 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:07.913753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:07.913729 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:08.550887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.550338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerStarted","Data":"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404"} Oct 30 00:50:08.555849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.555312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerStarted","Data":"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483"} Oct 30 00:50:08.611478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.606298 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:08.614895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.611843 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:08.614895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.611962 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:08.943396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:08.943361 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:09.037030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.036980 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:50:09.037428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.037404 5938 patch_prober.go:29] interesting pod/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.037613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.037587 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 00:50:09.496928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.496854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:09.613886 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:09.587394 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:09.628887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.617265 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:09.628887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.622025 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:09.649508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.649459 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:09.745925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.738293 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg"] Oct 30 00:50:09.764551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.764426 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba85458-v7hcg"] Oct 30 00:50:09.822915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.822699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:09.868441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.868407 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:50:09.883799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.883763 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7606b5-8833-494a-9670-24e1253bc8a3" path="/var/lib/kubelet/pods/ad7606b5-8833-494a-9670-24e1253bc8a3/volumes" Oct 30 00:50:09.961142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:09.961097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:10.045970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.029408 5938 patch_prober.go:29] interesting pod/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.045970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.029482 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 00:50:10.566296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.566264 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:10.609111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.609076 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:10.609406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.609375 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:10.658881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.658825 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" containerID="cri-o://5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483" gracePeriod=30 Oct 30 00:50:10.746364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.746314 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:10.921799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:10.921720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:11.058969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.058930 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:50:11.060049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.060019 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ad7606b5-8833-494a-9670-24e1253bc8a3" containerName="revoke-keys" Oct 30 00:50:11.060049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.060049 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7606b5-8833-494a-9670-24e1253bc8a3" containerName="revoke-keys" Oct 30 00:50:11.060271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.060255 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7606b5-8833-494a-9670-24e1253bc8a3" containerName="revoke-keys" Oct 30 00:50:11.064018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.063995 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.065222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.065199 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:50:11.071322 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.071299 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz"] Oct 30 00:50:11.077062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.077039 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e9dc7d-zxqqz"] Oct 30 00:50:11.079577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.079561 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-751d6789-openshift-default-dockercfg-mqjs2\"" Oct 30 00:50:11.133386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.133353 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:11.240764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.240681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.240925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.240828 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.240925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.240851 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.240923 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.240948 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.241008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbvj\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.241044 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.241061 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.241171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.241077 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.312282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.312249 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350159 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350225 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350333 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbvj\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350397 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350508 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350731 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.350765 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.351771 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.384922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.372210 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.384922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.372467 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.384922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.373030 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.384922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.373654 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.402592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.402556 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.421262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.421230 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.471886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.466432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.522948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.516818 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbvj\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\") pod \"e2e-ait-751d6789-openshift-default-986c6d7-lfbwt\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.527459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.523215 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 00:50:11.577720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.569673 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.581002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.580971 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.587941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.588017 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.588076 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.588197 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.588267 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.597640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.588329 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-959kp\" (UniqueName: \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.601107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.601079 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"kube-root-ca.crt\"" Oct 30 00:50:11.602056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.602035 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"default-dockercfg-vwhx2\"" Oct 30 00:50:11.602300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.602283 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"distinct-2-w1-29be73-kserve-self-signed-certs\"" Oct 30 00:50:11.602544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.602528 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"openshift-service-ca.crt\"" Oct 30 00:50:11.613889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.610200 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:11.618012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.617980 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 00:50:11.623516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.616496 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:11.727437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.727400 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:11.770103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.770049 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.770991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.770960 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.772803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.772778 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.773129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.773070 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.773310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.773294 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.774740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.774713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959kp\" (UniqueName: \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.811021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.804770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.819885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.815329 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:11.943210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:11.923213 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:12.026943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.026649 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:12.129590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.126337 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:12.130117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.129700 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36353c1c-1c47-4bd9-804b-fe862dcbb322" path="/var/lib/kubelet/pods/36353c1c-1c47-4bd9-804b-fe862dcbb322/volumes" Oct 30 00:50:12.149579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.149540 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29"] Oct 30 00:50:12.154220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.154191 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c498d9c-76s29"] Oct 30 00:50:12.154411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.154386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-959kp\" (UniqueName: \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\") pod \"distinct-2-w1-29be73-kserve-869dbf6b89-r49mt\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:12.431887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.431797 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:12.453303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.449828 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:50:12.460355 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:50:12.460324 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788989e3_39e9_4b3c_a2e2_e9e75aed847b.slice/crio-deff7e60475613dd1be386863d5a7754319a5e2fe23178c306f46855f12de124 WatchSource:0}: Error finding container deff7e60475613dd1be386863d5a7754319a5e2fe23178c306f46855f12de124: Status 404 returned error can't find the container with id deff7e60475613dd1be386863d5a7754319a5e2fe23178c306f46855f12de124 Oct 30 00:50:12.464996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.464963 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:12.465099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.465025 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:12.465099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.465059 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:12.638916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.632097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:12.638916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.632429 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:12.638916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.632476 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:12.676324 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:12.673841 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:12.710940 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:12.710593 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:12.875527 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.875459 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerStarted","Data":"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed"} Oct 30 00:50:12.875527 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.875516 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerStarted","Data":"deff7e60475613dd1be386863d5a7754319a5e2fe23178c306f46855f12de124"} Oct 30 00:50:12.948948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:12.945792 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podStartSLOduration=2.945772222 podStartE2EDuration="2.945772222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:12.938298705 +0000 UTC m=+1065.178533823" watchObservedRunningTime="2026-10-30 00:50:12.945772222 +0000 UTC m=+1065.186007336" Oct 30 00:50:13.047991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.047944 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 00:50:13.610854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.608828 5938 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 00:50:13.610854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.608908 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 00:50:13.750523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.750486 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:13.755342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.755314 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:13.877762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.877652 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322c8f9a-8f06-40c9-84c4-9f13675d5aa1" path="/var/lib/kubelet/pods/322c8f9a-8f06-40c9-84c4-9f13675d5aa1/volumes" Oct 30 00:50:13.909897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.908930 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" event={"ID":"c3487d6f-772d-4943-8244-0b2690c4b0f5","Type":"ContainerStarted","Data":"0b48937cd557caa042d1ceb1f004f4b3ccc3b81642346a700bac99fcc0583b49"} Oct 30 00:50:13.909897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.908980 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:13.909897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.909012 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" event={"ID":"c3487d6f-772d-4943-8244-0b2690c4b0f5","Type":"ContainerStarted","Data":"ef354f844382adbe18f8c2a5a10b27d027a0b03cdae13f2c01bdcb8dddef3aa0"} Oct 30 00:50:13.912136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.912076 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:50:13.961136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:13.961086 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podStartSLOduration=2.961069305 podStartE2EDuration="2.961069305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:13.956723596 +0000 UTC m=+1066.196958709" watchObservedRunningTime="2026-10-30 00:50:13.961069305 +0000 UTC m=+1066.201304446" Oct 30 00:50:14.064737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.064695 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:14.459882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.456794 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:14.621568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.621530 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:14.774707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.774623 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:14.775951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.775932 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:14.777772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.777748 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:14.788858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.788830 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:14.819024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.819002 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:50:14.940010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:14.939972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:15.089089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.088776 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:15.092503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.092464 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" containerID="cri-o://ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72" gracePeriod=30 Oct 30 00:50:15.124424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.123795 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:50:15.234905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.233358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:15.333888 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:15.330375 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:15.827764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.827725 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:15.926199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:15.926162 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" containerID="cri-o://9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404" gracePeriod=30 Oct 30 00:50:16.212768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.212734 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:50:16.217503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.217478 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.237469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.237445 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-be862c05-openshift-default-dockercfg-5t5dk\"" Oct 30 00:50:16.237926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.237906 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:50:16.259226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259199 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259242 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259444 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259499 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259604 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259704 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.259878 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.259846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxjk\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.260035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.260017 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.307431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.307407 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:16.361005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.360975 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361055 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361122 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361195 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361226 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxjk\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361336 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361439 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361512 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361536 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361814 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.361994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.361987 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.363496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.363474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.363600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.363583 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.383616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.383593 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.383857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.383839 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxjk\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\") pod \"e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.460402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.460381 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:16.534797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.534735 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:16.844065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:16.843556 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:17.014621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.014574 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:50:17.021610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.021560 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:17.021717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.021664 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:17.021759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.021711 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:17.944465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.944430 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerStarted","Data":"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b"} Oct 30 00:50:17.944847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:17.944480 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerStarted","Data":"5e23738e7e70dabe99a9d8d4c4a221300e489062143cd5e90ec19e755c68af24"} Oct 30 00:50:18.009570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.009056 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podStartSLOduration=2.00903791 podStartE2EDuration="2.00903791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:18.007337741 +0000 UTC m=+1070.247572856" watchObservedRunningTime="2026-10-30 00:50:18.00903791 +0000 UTC m=+1070.249273024" Oct 30 00:50:18.530788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.530746 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:18.535668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.535621 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:18.545858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.545833 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:18.954613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.954581 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:18.955520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:18.955496 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:50:19.692351 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:19.692319 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice/crio-45e712676562decea2de4c4e4cb359b1945a457f3b203978be4b9e7d3bec88de\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaee8520_fb23_4258_9ff2_a2e1510eb145.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:50:19.867123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:19.867091 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:20.003130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.003058 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:20.330895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.330839 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:20.464124 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.464102 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519356 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519452 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519521 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519678 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519743 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519874 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519927 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.519987 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.520104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.520027 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\") pod \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\" (UID: \"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc\") " Oct 30 00:50:20.521030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.521001 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.521278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.521254 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.521502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.521480 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.521577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.521496 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.521983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.521954 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:50:20.525844 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.525810 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "kube-api-access-sjlt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:20.526147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.526113 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:20.526276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.526245 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:50:20.530249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.530223 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy" pod "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" (UID: "0d6da20e-3e2d-48e4-b5ea-0274ab0869dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.592261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.592231 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:50:20.593158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.593125 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" Oct 30 00:50:20.593158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.593150 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" Oct 30 00:50:20.593430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.593357 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerName="istio-proxy" Oct 30 00:50:20.598407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.598388 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.618813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.618786 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623449 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623482 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623495 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623510 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623526 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623540 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjlt7\" (UniqueName: \"kubernetes.io/projected/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-kube-api-access-sjlt7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623554 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.623590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.623569 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.640093 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.640072 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:50:20.724291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724305 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724341 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724469 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724583 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.724662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724623 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724671 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724697 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.724948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.724770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\") pod \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\" (UID: \"ca72d5aa-2556-4ddc-beab-0b8225de79ec\") " Oct 30 00:50:20.725131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725140 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.725222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725163 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725505 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgngv\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725570 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725618 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.725804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725786 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725802 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.725970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725927 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.725970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725961 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.726089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.725989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.726417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.726394 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.726509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.726421 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.726509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.726436 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.726509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.726450 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.727397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.727367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:50:20.728318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.728291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:50:20.728690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.728668 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:20.728958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.728935 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "kube-api-access-6w77r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:20.729051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.728976 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token" pod "ca72d5aa-2556-4ddc-beab-0b8225de79ec" (UID: "ca72d5aa-2556-4ddc-beab-0b8225de79ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:20.828349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828233 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828318 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828339 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828361 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828475 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828513 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828558 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgngv\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828749 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.828881 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828885 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828913 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828934 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828931 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828948 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.828982 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w77r\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-kube-api-access-6w77r\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.829067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.829005 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca72d5aa-2556-4ddc-beab-0b8225de79ec-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:20.829402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.829136 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.829560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.829524 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.830816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.830795 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.831373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.831352 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.853308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.853286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.853761 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.853738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgngv\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.923136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.923082 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:20.982020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.981979 5938 generic.go:356] "Generic (PLEG): container finished" podID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerID="5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483" exitCode=0 Oct 30 00:50:20.982207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.982059 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" Oct 30 00:50:20.982207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.982062 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerDied","Data":"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483"} Oct 30 00:50:20.982207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.982130 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq" event={"ID":"ca72d5aa-2556-4ddc-beab-0b8225de79ec","Type":"ContainerDied","Data":"f1db445840f1db91fde052a4934b0b8ee10a2eca5fb3bff4c1c93e749809ebba"} Oct 30 00:50:20.982207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.982163 5938 scope.go:122] "RemoveContainer" containerID="5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483" Oct 30 00:50:20.984248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.984227 5938 generic.go:356] "Generic (PLEG): container finished" podID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" containerID="ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72" exitCode=0 Oct 30 00:50:20.984354 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.984317 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" event={"ID":"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc","Type":"ContainerDied","Data":"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72"} Oct 30 00:50:20.984402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.984369 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" event={"ID":"0d6da20e-3e2d-48e4-b5ea-0274ab0869dc","Type":"ContainerDied","Data":"f052f95267816efe5794268f70ac54d0aee539b02645f71f34fc118ac7b0d960"} Oct 30 00:50:20.984402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:20.984326 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf" Oct 30 00:50:21.047630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.047606 5938 scope.go:122] "RemoveContainer" containerID="2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6" Oct 30 00:50:21.048795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.048767 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx"] Oct 30 00:50:21.072584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.055611 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt"] Oct 30 00:50:21.072584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.064031 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a845fb-h66gx"] Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.073429 5938 scope.go:122] "RemoveContainer" containerID="5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:21.073811 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483\": container with ID starting with 5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483 not found: ID does not exist" containerID="5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.073850 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483"} err="failed to get container status \"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483\": rpc error: code = NotFound desc = could not find container \"5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483\": container with ID starting with 5aacb2f023e6e9204e3ac5f94972a79ae7df09cb3e1fe30fc31b686c84c88483 not found: ID does not exist" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.073930 5938 scope.go:122] "RemoveContainer" containerID="2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:21.074270 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6\": container with ID starting with 2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6 not found: ID does not exist" containerID="2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.074301 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6"} err="failed to get container status \"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6\": rpc error: code = NotFound desc = could not find container \"2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6\": container with ID starting with 2aa7261f926606d30c7f48517bd71b76cfe70341428213e48c2c68f9f2947af6 not found: ID does not exist" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.074323 5938 scope.go:122] "RemoveContainer" containerID="ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72" Oct 30 00:50:21.081330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.076026 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d470fe-l48bt"] Oct 30 00:50:21.090809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.084461 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:21.095516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.095406 5938 scope.go:122] "RemoveContainer" containerID="ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72" Oct 30 00:50:21.095900 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:21.095805 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72\": container with ID starting with ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72 not found: ID does not exist" containerID="ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72" Oct 30 00:50:21.095900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.095845 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-b8cmf"] Oct 30 00:50:21.095900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.095840 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72"} err="failed to get container status \"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72\": rpc error: code = NotFound desc = could not find container \"ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72\": container with ID starting with ea82c0b81853c9b7b85d47bd60c03836899b8eed695d254cf0e0e92c56a0ab72 not found: ID does not exist" Oct 30 00:50:21.111966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.105304 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:50:21.112694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.112668 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ba85458-openshift-default-5f56f8659c-fsdfq"] Oct 30 00:50:21.273811 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:50:21.273781 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24a1159_1d8d_48dd_bbac_78f509c62976.slice/crio-19ef8b772d245a172774f7d2f033d012a583224c01bfae5f9482e728311d5273 WatchSource:0}: Error finding container 19ef8b772d245a172774f7d2f033d012a583224c01bfae5f9482e728311d5273: Status 404 returned error can't find the container with id 19ef8b772d245a172774f7d2f033d012a583224c01bfae5f9482e728311d5273 Oct 30 00:50:21.275963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.275938 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:50:21.277054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.277019 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.277150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.277121 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.277200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.277165 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:21.287902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.286799 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:21.341132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341100 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341203 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341275 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341374 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341422 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:21.341454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341429 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341631 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:21.341726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341643 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341781 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341892 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\") pod \"84070897-5bc1-426b-ba2f-884efde7bc03\" (UID: \"84070897-5bc1-426b-ba2f-884efde7bc03\") " Oct 30 00:50:21.341977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.341947 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:21.342077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342052 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:21.342358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342333 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:50:21.342756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342736 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.342841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342760 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.342841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342774 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.342841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342806 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84070897-5bc1-426b-ba2f-884efde7bc03-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.342841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.342821 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.343572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.343542 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:21.344068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.344045 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:50:21.344197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.344179 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "kube-api-access-sxzdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:21.344667 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.344648 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo" pod "84070897-5bc1-426b-ba2f-884efde7bc03" (UID: "84070897-5bc1-426b-ba2f-884efde7bc03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:50:21.447619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.447517 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84070897-5bc1-426b-ba2f-884efde7bc03-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.447619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.447551 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84070897-5bc1-426b-ba2f-884efde7bc03-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.447619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.447567 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.447619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.447582 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxzdm\" (UniqueName: \"kubernetes.io/projected/84070897-5bc1-426b-ba2f-884efde7bc03-kube-api-access-sxzdm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:21.720842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.720799 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:21.872830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.872791 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:21.878745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.878716 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6da20e-3e2d-48e4-b5ea-0274ab0869dc" path="/var/lib/kubelet/pods/0d6da20e-3e2d-48e4-b5ea-0274ab0869dc/volumes" Oct 30 00:50:21.879342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.879319 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1199f95a-06a7-41c1-bf0e-876eaf9790b8" path="/var/lib/kubelet/pods/1199f95a-06a7-41c1-bf0e-876eaf9790b8/volumes" Oct 30 00:50:21.879894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.879876 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" path="/var/lib/kubelet/pods/ca72d5aa-2556-4ddc-beab-0b8225de79ec/volumes" Oct 30 00:50:21.880770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:21.880750 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ed0d70-32a6-4019-bbf1-898a55429bc5" path="/var/lib/kubelet/pods/f8ed0d70-32a6-4019-bbf1-898a55429bc5/volumes" Oct 30 00:50:22.089057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.088955 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p"] Oct 30 00:50:22.181888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.176682 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8cda2a-cbl9p"] Oct 30 00:50:22.261583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.261416 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:22.457916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.456439 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:22.502990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.491206 5938 generic.go:356] "Generic (PLEG): container finished" podID="84070897-5bc1-426b-ba2f-884efde7bc03" containerID="9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404" exitCode=0 Oct 30 00:50:22.502990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.491319 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerDied","Data":"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404"} Oct 30 00:50:22.502990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.491370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" event={"ID":"84070897-5bc1-426b-ba2f-884efde7bc03","Type":"ContainerDied","Data":"6af812213f654f68f0e6b66615b15c891f1e5b2641010dc798e63d096682546e"} Oct 30 00:50:22.502990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.491405 5938 scope.go:122] "RemoveContainer" containerID="9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404" Oct 30 00:50:22.502990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.491530 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s" Oct 30 00:50:22.519805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.519285 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" event={"ID":"b24a1159-1d8d-48dd-bbac-78f509c62976","Type":"ContainerStarted","Data":"bee3b86c676b66d70fcb1c878ab6d4a9b5854e5b4fff3d33a03ca686c89195ae"} Oct 30 00:50:22.519805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.519347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" event={"ID":"b24a1159-1d8d-48dd-bbac-78f509c62976","Type":"ContainerStarted","Data":"19ef8b772d245a172774f7d2f033d012a583224c01bfae5f9482e728311d5273"} Oct 30 00:50:22.592251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.591158 5938 scope.go:122] "RemoveContainer" containerID="6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef" Oct 30 00:50:22.644713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.644686 5938 scope.go:122] "RemoveContainer" containerID="9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404" Oct 30 00:50:22.648938 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:22.648152 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404\": container with ID starting with 9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404 not found: ID does not exist" containerID="9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404" Oct 30 00:50:22.648938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.648197 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404"} err="failed to get container status \"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404\": rpc error: code = NotFound desc = could not find container \"9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404\": container with ID starting with 9ae37291dd031760a516d47ef9a825c15373c4b7f62a2f2dae42adabb69d9404 not found: ID does not exist" Oct 30 00:50:22.648938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.648224 5938 scope.go:122] "RemoveContainer" containerID="6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef" Oct 30 00:50:22.649937 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:22.649911 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef\": container with ID starting with 6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef not found: ID does not exist" containerID="6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef" Oct 30 00:50:22.650071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.650049 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef"} err="failed to get container status \"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef\": rpc error: code = NotFound desc = could not find container \"6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef\": container with ID starting with 6cf1867548c2ae5b71978a757c3f2bb5c4983fe371727a9a6355ba74df6e82ef not found: ID does not exist" Oct 30 00:50:22.662010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.661472 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:22.668231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.668171 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podStartSLOduration=2.668152852 podStartE2EDuration="2.668152852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:22.582175228 +0000 UTC m=+1074.822410345" watchObservedRunningTime="2026-10-30 00:50:22.668152852 +0000 UTC m=+1074.908387968" Oct 30 00:50:22.808014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.807921 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:50:22.925884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.923644 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:22.927942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.926199 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-mfs2s"] Oct 30 00:50:22.928196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.928172 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:22.928376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:22.928354 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:23.932886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:23.931965 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:23.932886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:23.932042 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:23.933918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:23.933738 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f37b6a2-bdf4-4e07-b27c-e694d9576680" path="/var/lib/kubelet/pods/0f37b6a2-bdf4-4e07-b27c-e694d9576680/volumes" Oct 30 00:50:23.934901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:23.934515 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" path="/var/lib/kubelet/pods/84070897-5bc1-426b-ba2f-884efde7bc03/volumes" Oct 30 00:50:23.938901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:23.935059 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:24.392360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:24.391924 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:24.416888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:24.412006 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:50:24.923706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:24.923674 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:24.923989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:24.923734 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:25.001339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.001305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:25.033699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.033667 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt"] Oct 30 00:50:25.044707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.044661 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d962fca6-cx8zt"] Oct 30 00:50:25.877746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.877702 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24395531-dc30-4ca6-938f-1b153b905b19" path="/var/lib/kubelet/pods/24395531-dc30-4ca6-938f-1b153b905b19/volumes" Oct 30 00:50:25.924429 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.924392 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:25.924573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:25.924459 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:26.029958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:26.029924 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf"] Oct 30 00:50:26.041439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:26.041412 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6eaa96-h29wf"] Oct 30 00:50:26.674051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:26.674017 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:26.924191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:26.924112 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:26.924191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:26.924159 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:27.877489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:27.877455 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978c87ae-fcc4-4324-b0f8-d27a03867812" path="/var/lib/kubelet/pods/978c87ae-fcc4-4324-b0f8-d27a03867812/volumes" Oct 30 00:50:27.923657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:27.923629 5938 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 00:50:27.923822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:27.923694 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 00:50:28.250064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:28.249959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:28.839035 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:28.839002 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:50:29.268114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:29.265539 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:29.268114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:29.267238 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:29.274009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:29.273940 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:50:30.048208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:30.048162 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:30.189586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:30.189548 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:30.346632 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:30.346537 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:31.762652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:31.762619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:33.453019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:33.452969 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.232738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:34.232709 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.376800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:34.376762 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.540403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:34.540318 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:35.035848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.035811 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj"] Oct 30 00:50:35.048453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.048420 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9ce1bd-54knj"] Oct 30 00:50:35.133934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.133893 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:35.270778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.270748 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:35.563571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.563534 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:35.877222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.877184 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511bfec0-5b99-4c6a-a9bc-da6582179cbb" path="/var/lib/kubelet/pods/511bfec0-5b99-4c6a-a9bc-da6582179cbb/volumes" Oct 30 00:50:35.917782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.917754 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:50:35.918386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918370 5938 cpu_manager.go:414] "containerMap: removing container" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:35.918386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918386 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:35.918514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918404 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:35.918514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918409 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:35.918593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918529 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:35.918593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918541 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:35.918593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.918549 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:35.922827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.922811 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:35.958673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.957970 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:50:35.997014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.996975 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:35.997378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.997290 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:35.999830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:35.999787 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000074 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000522 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000595 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kw8m\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000683 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.000763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.000741 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102176 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102229 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102324 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kw8m\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102447 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102502 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102581 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102599 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102637 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.102951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.103005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.102964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.103098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.103081 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.104628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.104606 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.104939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.104921 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.125013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.124990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.125205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.125184 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kw8m\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.222891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.222806 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:36.236258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.236232 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:36.585843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.585804 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:50:36.586712 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:50:36.586679 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce8ba35_7602_4ebb_b1d6_8a0fbe5f35cc.slice/crio-8d78b4748208b83d7c8a13199b3d5b0b35233c1f16d1db737ed8f37bca400530 WatchSource:0}: Error finding container 8d78b4748208b83d7c8a13199b3d5b0b35233c1f16d1db737ed8f37bca400530: Status 404 returned error can't find the container with id 8d78b4748208b83d7c8a13199b3d5b0b35233c1f16d1db737ed8f37bca400530 Oct 30 00:50:36.591824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.591761 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:36.591962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.591889 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:36.591962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.591947 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:36.764384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.763137 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" event={"ID":"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc","Type":"ContainerStarted","Data":"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e"} Oct 30 00:50:36.764384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.763196 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" event={"ID":"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc","Type":"ContainerStarted","Data":"8d78b4748208b83d7c8a13199b3d5b0b35233c1f16d1db737ed8f37bca400530"} Oct 30 00:50:36.814500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:36.813564 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podStartSLOduration=1.81354622 podStartE2EDuration="1.81354622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:36.806673791 +0000 UTC m=+1089.046908927" watchObservedRunningTime="2026-10-30 00:50:36.81354622 +0000 UTC m=+1089.053781333" Oct 30 00:50:37.222978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:37.222939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:37.236662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:37.236631 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:37.238777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:37.238747 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:37.238955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:37.238808 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:38.236544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:38.236510 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:38.237032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:38.236564 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:38.940707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:38.940614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:39.236950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:39.236591 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:39.236950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:39.236652 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:40.236943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:40.236526 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:40.236943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:40.236591 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:40.560048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:40.559960 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:40.722936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:40.722905 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:40.974148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:40.974118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:41.132919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:41.132851 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:41.236630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:41.236539 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:41.236802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:41.236641 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:42.236539 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:42.236505 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:42.236997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:42.236560 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:42.375487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:42.375454 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:43.236879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:43.236845 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 00:50:43.237286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:43.236907 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 00:50:43.977359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:43.977299 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:44.240688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:44.240616 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:44.241172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:44.241154 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:44.241992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:44.241969 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:50:45.559926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:45.559853 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:47.030898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:47.030856 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c"] Oct 30 00:50:47.042650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:47.042626 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a45792a-jmj4c"] Oct 30 00:50:47.255220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:47.255183 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:47.400673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:47.400611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:47.880041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:47.880011 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed01342-41b2-4c67-b09e-a05dc6c10ba4" path="/var/lib/kubelet/pods/6ed01342-41b2-4c67-b09e-a05dc6c10ba4/volumes" Oct 30 00:50:48.568794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:48.568732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:48.919915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:48.916246 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:49.239659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.239570 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 00:50:49.251892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.240797 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:49.252135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.252113 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:49.268574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.268542 5938 cpu_manager.go:414] "containerMap: removing container" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:49.268768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.268752 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84070897-5bc1-426b-ba2f-884efde7bc03" containerName="istio-proxy" Oct 30 00:50:49.269181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.269150 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca72d5aa-2556-4ddc-beab-0b8225de79ec" containerName="istio-proxy" Oct 30 00:50:49.370147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.370118 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 00:50:49.370387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.370224 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:49.370509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.370329 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.400054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.400030 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-11f521-openshift-default-dockercfg-vkflh\"" Oct 30 00:50:49.498286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498190 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498369 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498509 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h778\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498550 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498661 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498788 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.498917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.498895 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.499071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.499053 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.499142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.499094 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601076 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601158 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601188 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601377 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601474 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h778\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601536 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.601716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601570 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.602090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.601933 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.602741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.602711 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.602903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.602763 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.602903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.602764 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.602903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.602832 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.604832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.604802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.605015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.604996 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.624566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.624533 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.627990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.627964 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h778\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.701483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.701445 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:49.731986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:49.731944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:50.126044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:50.125994 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:50.126371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:50.126343 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:50.126624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:50.126599 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:50:50.127724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:50.127701 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 00:50:51.087327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.087267 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerStarted","Data":"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5"} Oct 30 00:50:51.087857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.087344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerStarted","Data":"c088e9d7a2d84a4b857f789dd0db88ba620f83e547b7be2fe015fb1a09b11cad"} Oct 30 00:50:51.175884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.174552 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podStartSLOduration=2.174529614 podStartE2EDuration="2.174529614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:51.151343097 +0000 UTC m=+1103.391578223" watchObservedRunningTime="2026-10-30 00:50:51.174529614 +0000 UTC m=+1103.414764728" Oct 30 00:50:51.228904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.223825 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs"] Oct 30 00:50:51.269900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.255023 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.273899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.272735 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:50:51.273899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.273430 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:50:51.273899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.273528 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:50:51.281723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.281107 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:50:51.506541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.481474 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.506541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.481665 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.506541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.481776 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5qq\" (UniqueName: \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.646970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.613189 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.646970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.613350 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5qq\" (UniqueName: \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.646970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.613609 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.646970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.614700 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.705107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.680127 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:51.731699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:51.731649 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:52.126929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.013261 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:52.126929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.084611 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5qq\" (UniqueName: \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\") pod \"maas-api-revoke-keys-e2e-ait-751d6789-nxlcs\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:52.127580 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.127540 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.146161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.145848 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:52.181238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.181162 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs"] Oct 30 00:50:52.181419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.181340 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:50:52.298890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.282788 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.324887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.319972 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:50:52.401945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.397490 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.421105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.410578 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.421105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.411018 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:50:52.421105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.411303 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:50:52.421105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.411472 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:50:52.421105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.415435 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.430971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.430938 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:52.445460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.438584 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:52.445749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.439050 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:50:52.493071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.492654 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:50:52.754948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.753573 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:52.904998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:52.904941 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": context deadline exceeded" Oct 30 00:50:53.304904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.251395 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:50:53.304904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.251745 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:53.304904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.252484 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:50:53.304904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.252542 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:50:53.304904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.268172 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 00:50:53.443957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.434951 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 00:50:53.443957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.435426 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:50:53.468887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.444659 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:50:53.468887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.444738 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:50:53.522241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.522197 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/health\": context deadline exceeded" Oct 30 00:50:53.534913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.523390 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:50:53.576853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.576813 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:50:53.577199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.577172 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:50:53.578107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.578086 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:50:53.578986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.578969 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:50:53.579180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.579166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:50:53.747224 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.730365 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:53.752378 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.747416 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:50:53.752901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.752883 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:50:53.786883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.771688 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:50:53.787462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.747450 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:50:53.788532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.751733 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:53.788532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.788234 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:50:53.870971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.870929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:53.873476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:53.871656 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 00:50:54.063433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.063229 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 00:50:54.064835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.064786 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:54.133754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.133677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:54.174829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.172623 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:54.379969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.284780 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:50:54.382060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.320040 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:54.398890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.368366 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:54.398890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.382325 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:50:54.398890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.398850 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:54.399148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.398958 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:50:54.399148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.399094 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 00:50:54.399266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.399166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:50:54.476328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.476305 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:50:54.477149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.477118 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:50:54.477258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.477189 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:50:54.478149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.478122 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:50:54.579965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.568732 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs"] Oct 30 00:50:54.651981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.646744 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 00:50:54.664417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.664116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:50:54.765692 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:54.764471 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" event={"ID":"798a3fdc-2a1b-49e5-a6d8-1027f982d820","Type":"ContainerStarted","Data":"7e2510916de729d3baa3abafc88b34119042d777b0ad85282675ff5347e8f0c4"} Oct 30 00:50:55.451810 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:55.451733 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.626831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:55.626733 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:56.147988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:56.089740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" event={"ID":"798a3fdc-2a1b-49e5-a6d8-1027f982d820","Type":"ContainerStarted","Data":"239cc0dc3f337722bac3712621c6da21f6e878a53439076e4e19574c8bf3d727"} Oct 30 00:50:56.147988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:56.091284 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:56.163999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:56.163938 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" podStartSLOduration=5.163910307 podStartE2EDuration="5.163910307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:56.163546443 +0000 UTC m=+1108.403781558" watchObservedRunningTime="2026-10-30 00:50:56.163910307 +0000 UTC m=+1108.404152817" Oct 30 00:50:57.145048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:57.145011 5938 generic.go:356] "Generic (PLEG): container finished" podID="798a3fdc-2a1b-49e5-a6d8-1027f982d820" containerID="239cc0dc3f337722bac3712621c6da21f6e878a53439076e4e19574c8bf3d727" exitCode=0 Oct 30 00:50:57.145533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:57.145088 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" event={"ID":"798a3fdc-2a1b-49e5-a6d8-1027f982d820","Type":"ContainerDied","Data":"239cc0dc3f337722bac3712621c6da21f6e878a53439076e4e19574c8bf3d727"} Oct 30 00:50:58.009048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.009004 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:58.339424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.339396 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:58.474422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.474391 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\" (UniqueName: \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\") pod \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " Oct 30 00:50:58.474606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.474527 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\") pod \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " Oct 30 00:50:58.474786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.474755 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\") pod \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\" (UID: \"798a3fdc-2a1b-49e5-a6d8-1027f982d820\") " Oct 30 00:50:58.475194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.475161 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca" pod "798a3fdc-2a1b-49e5-a6d8-1027f982d820" (UID: "798a3fdc-2a1b-49e5-a6d8-1027f982d820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:50:58.475457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.475439 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:58.476786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.476762 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq" pod "798a3fdc-2a1b-49e5-a6d8-1027f982d820" (UID: "798a3fdc-2a1b-49e5-a6d8-1027f982d820"). InnerVolumeSpecName "kube-api-access-rz5qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:50:58.477097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.477076 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca" pod "798a3fdc-2a1b-49e5-a6d8-1027f982d820" (UID: "798a3fdc-2a1b-49e5-a6d8-1027f982d820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:50:58.576948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.576800 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5qq\" (UniqueName: \"kubernetes.io/projected/798a3fdc-2a1b-49e5-a6d8-1027f982d820-kube-api-access-rz5qq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:58.576948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:58.576832 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798a3fdc-2a1b-49e5-a6d8-1027f982d820-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:50:58.835978 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:50:58.835875 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:50:59.173640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.169925 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" Oct 30 00:50:59.173640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.169942 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs" event={"ID":"798a3fdc-2a1b-49e5-a6d8-1027f982d820","Type":"ContainerDied","Data":"7e2510916de729d3baa3abafc88b34119042d777b0ad85282675ff5347e8f0c4"} Oct 30 00:50:59.173640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.170008 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2510916de729d3baa3abafc88b34119042d777b0ad85282675ff5347e8f0c4" Oct 30 00:50:59.586713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.586620 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:59.949055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.949018 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:50:59.949232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:50:59.949064 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:00.293230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.293158 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg"] Oct 30 00:51:00.293834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.293820 5938 cpu_manager.go:414] "containerMap: removing container" podUID="798a3fdc-2a1b-49e5-a6d8-1027f982d820" containerName="revoke-keys" Oct 30 00:51:00.293926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.293835 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798a3fdc-2a1b-49e5-a6d8-1027f982d820" containerName="revoke-keys" Oct 30 00:51:00.293973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.293964 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="798a3fdc-2a1b-49e5-a6d8-1027f982d820" containerName="revoke-keys" Oct 30 00:51:00.351303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.351210 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg"] Oct 30 00:51:00.351303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.351268 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.381569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.381541 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:51:00.381699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.381541 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:00.391092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.391070 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:51:00.391205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.391146 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:51:00.441000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.440968 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.441150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.441016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f52k\" (UniqueName: \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.441150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.441035 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.542903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.542847 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.543106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.542939 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f52k\" (UniqueName: \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.543106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.542963 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.543920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.543810 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.545309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.545286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.565753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.565722 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f52k\" (UniqueName: \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\") pod \"maas-api-revoke-keys-e2e-derive-be862c05-km4zg\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:00.743673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:00.743635 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:01.181210 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:01.181176 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3805acf5_5517_47f2_9556_793c2e7b9eed.slice/crio-6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0 WatchSource:0}: Error finding container 6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0: Status 404 returned error can't find the container with id 6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0 Oct 30 00:51:01.181335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:01.181242 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg"] Oct 30 00:51:01.289631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:01.289595 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:01.881325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:01.881282 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:02.023355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:02.023300 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:02.203984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:02.203911 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" event={"ID":"3805acf5-5517-47f2-9556-793c2e7b9eed","Type":"ContainerStarted","Data":"fed0f80a2433ba4568f29ac0242d94700396beb7c0b440c2f6be6501ddbf588f"} Oct 30 00:51:02.203984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:02.203979 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" event={"ID":"3805acf5-5517-47f2-9556-793c2e7b9eed","Type":"ContainerStarted","Data":"6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0"} Oct 30 00:51:02.247565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:02.243428 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" podStartSLOduration=2.243409763 podStartE2EDuration="2.243409763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:02.242679805 +0000 UTC m=+1114.482914919" watchObservedRunningTime="2026-10-30 00:51:02.243409763 +0000 UTC m=+1114.483644879" Oct 30 00:51:03.119256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.119183 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:03.228188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.228152 5938 generic.go:356] "Generic (PLEG): container finished" podID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerID="b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed" exitCode=0 Oct 30 00:51:03.228386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.228230 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerDied","Data":"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed"} Oct 30 00:51:03.228744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.228716 5938 scope.go:122] "RemoveContainer" containerID="b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed" Oct 30 00:51:03.230791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.230747 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:03.230933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.230844 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:03.230933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.230908 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:03.230933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.230917 5938 generic.go:356] "Generic (PLEG): container finished" podID="3805acf5-5517-47f2-9556-793c2e7b9eed" containerID="fed0f80a2433ba4568f29ac0242d94700396beb7c0b440c2f6be6501ddbf588f" exitCode=0 Oct 30 00:51:03.231069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:03.231039 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" event={"ID":"3805acf5-5517-47f2-9556-793c2e7b9eed","Type":"ContainerDied","Data":"fed0f80a2433ba4568f29ac0242d94700396beb7c0b440c2f6be6501ddbf588f"} Oct 30 00:51:04.243113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.243075 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerStarted","Data":"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5"} Oct 30 00:51:04.515119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.515019 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:04.654927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.654895 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\" (UniqueName: \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\") pod \"3805acf5-5517-47f2-9556-793c2e7b9eed\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " Oct 30 00:51:04.655105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.655091 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\") pod \"3805acf5-5517-47f2-9556-793c2e7b9eed\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " Oct 30 00:51:04.655169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.655151 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\") pod \"3805acf5-5517-47f2-9556-793c2e7b9eed\" (UID: \"3805acf5-5517-47f2-9556-793c2e7b9eed\") " Oct 30 00:51:04.655771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.655692 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca" pod "3805acf5-5517-47f2-9556-793c2e7b9eed" (UID: "3805acf5-5517-47f2-9556-793c2e7b9eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:04.655985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.655963 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:04.657394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.657367 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k" pod "3805acf5-5517-47f2-9556-793c2e7b9eed" (UID: "3805acf5-5517-47f2-9556-793c2e7b9eed"). InnerVolumeSpecName "kube-api-access-4f52k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:04.657508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.657427 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca" pod "3805acf5-5517-47f2-9556-793c2e7b9eed" (UID: "3805acf5-5517-47f2-9556-793c2e7b9eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:51:04.750531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.750493 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:51:04.750810 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.750779 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:51:04.750914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.750848 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 00:51:04.757999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.757974 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f52k\" (UniqueName: \"kubernetes.io/projected/3805acf5-5517-47f2-9556-793c2e7b9eed-kube-api-access-4f52k\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:04.757999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.757996 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3805acf5-5517-47f2-9556-793c2e7b9eed-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:04.768899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.768807 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:04.956384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.956349 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:04.956552 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:04.956404 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:05.253549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.253516 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" Oct 30 00:51:05.253549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.253527 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg" event={"ID":"3805acf5-5517-47f2-9556-793c2e7b9eed","Type":"ContainerDied","Data":"6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0"} Oct 30 00:51:05.253985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.253576 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6649f8dee5433c4c9ec85d56cc50502ada6dd37d338e21bd071df332d4aac9a0" Oct 30 00:51:05.750610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.750575 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:51:05.750831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.750627 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 00:51:05.852165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.852129 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 00:51:05.852926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.852904 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3805acf5-5517-47f2-9556-793c2e7b9eed" containerName="revoke-keys" Oct 30 00:51:05.852926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.852924 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3805acf5-5517-47f2-9556-793c2e7b9eed" containerName="revoke-keys" Oct 30 00:51:05.853136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.853122 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3805acf5-5517-47f2-9556-793c2e7b9eed" containerName="revoke-keys" Oct 30 00:51:05.855605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.855585 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.927997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.927969 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 00:51:05.974543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974514 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974578 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974683 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974737 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974761 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974914 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2br76\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.974973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.974952 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.975050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.975039 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:05.975098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:05.975086 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076247 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br76\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076394 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076489 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076689 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076669 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076772 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076826 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076916 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.076978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.076956 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.077541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.077517 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.077719 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.077695 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.077996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.077972 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.078178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.078134 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.078748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.078726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.079975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.079930 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.080526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.080507 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.112213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.112183 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.115528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.115507 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.116375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.116352 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br76\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.118458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.118434 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt"] Oct 30 00:51:06.145063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.145019 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.152188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.152163 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt"] Oct 30 00:51:06.160525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.160505 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:51:06.160665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.160608 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:06.160948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.160674 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:51:06.161107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.160771 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:51:06.169618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.169597 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:06.283361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.283323 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.292893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.284128 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.309975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.309942 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46zx\" (UniqueName: \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.352015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.351982 5938 generic.go:356] "Generic (PLEG): container finished" podID="6113b106-66ff-403a-a697-f62657225c91" containerID="105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b" exitCode=0 Oct 30 00:51:06.352274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.352255 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerDied","Data":"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b"} Oct 30 00:51:06.352956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.352914 5938 scope.go:122] "RemoveContainer" containerID="105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b" Oct 30 00:51:06.356564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.355527 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.356564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.355616 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.356564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.355663 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.478917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.478628 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.478917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.478779 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46zx\" (UniqueName: \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.483694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.483652 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.535911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.525820 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46zx\" (UniqueName: \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.546297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.546259 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.551610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.551584 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8535811f-f4jzt\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.722128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.697931 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.815700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.795220 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:06.843387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.841992 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:51:06.843387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.842202 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 00:51:06.853329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.851578 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 00:51:06.853329 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:06.851674 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539ef6c2_f9e3_4032_ab01_f5d335399e64.slice/crio-94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f WatchSource:0}: Error finding container 94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f: Status 404 returned error can't find the container with id 94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f Oct 30 00:51:06.860047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.860006 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.860161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.860088 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.860161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.860130 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:06.892064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.892024 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.929635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.929603 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs"] Oct 30 00:51:06.936396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:06.936358 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-751d6789-nxlcs"] Oct 30 00:51:07.069477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.068420 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:07.224239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.224142 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt"] Oct 30 00:51:07.229047 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:07.229000 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50919b3_d60b_4b6e_94dc_7d54fd10cb1d.slice/crio-e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7 WatchSource:0}: Error finding container e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7: Status 404 returned error can't find the container with id e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7 Oct 30 00:51:07.366073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.366037 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerStarted","Data":"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1"} Oct 30 00:51:07.367755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.367728 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" event={"ID":"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d","Type":"ContainerStarted","Data":"e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7"} Oct 30 00:51:07.369786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.369761 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" event={"ID":"539ef6c2-f9e3-4032-ab01-f5d335399e64","Type":"ContainerStarted","Data":"83abc6044d62509766cf0f30947d3101948718ae9c1e8190ff00ce99d8d06baa"} Oct 30 00:51:07.369884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.369800 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" event={"ID":"539ef6c2-f9e3-4032-ab01-f5d335399e64","Type":"ContainerStarted","Data":"94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f"} Oct 30 00:51:07.460606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.460547 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podStartSLOduration=2.460532649 podStartE2EDuration="2.460532649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:07.459093151 +0000 UTC m=+1119.699328267" watchObservedRunningTime="2026-10-30 00:51:07.460532649 +0000 UTC m=+1119.700767761" Oct 30 00:51:07.536099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.536017 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:51:07.539975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.537969 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:07.549608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.540224 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:07.753413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.753365 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:51:07.755002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.754970 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 00:51:07.879066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:07.879028 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798a3fdc-2a1b-49e5-a6d8-1027f982d820" path="/var/lib/kubelet/pods/798a3fdc-2a1b-49e5-a6d8-1027f982d820/volumes" Oct 30 00:51:08.170364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.170282 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:08.172228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.172196 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:08.172460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.172432 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:08.474344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.474255 5938 generic.go:356] "Generic (PLEG): container finished" podID="b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" containerID="b88a53e46053f467c45fdd70d44e76dc33bacc26c41effb6225971b7a0a667a0" exitCode=0 Oct 30 00:51:08.474812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.474312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" event={"ID":"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d","Type":"ContainerDied","Data":"b88a53e46053f467c45fdd70d44e76dc33bacc26c41effb6225971b7a0a667a0"} Oct 30 00:51:08.765684 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.765596 5938 patch_prober.go:29] interesting pod/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 00:51:08.765962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.765930 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 00:51:08.792837 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.777555 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:08.792837 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.783640 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:08.792837 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.783702 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:08.947061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.947025 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:51:08.947573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:08.947538 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" containerID="cri-o://cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5" gracePeriod=30 Oct 30 00:51:09.089937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.089022 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:09.179356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.175640 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:09.179661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.179615 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:09.539899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.539717 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:09.540434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.539906 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:09.956197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:09.954652 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.054770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.054740 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:10.173487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.172560 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:10.173487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.172646 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:10.215790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.214771 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\") pod \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " Oct 30 00:51:10.215790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.214975 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\" (UniqueName: \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\") pod \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " Oct 30 00:51:10.215790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.215108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\") pod \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\" (UID: \"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d\") " Oct 30 00:51:10.216596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.216557 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca" pod "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" (UID: "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:10.265977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.265924 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx" pod "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" (UID: "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d"). InnerVolumeSpecName "kube-api-access-m46zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:10.266266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.266237 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca" pod "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" (UID: "b50919b3-d60b-4b6e-94dc-7d54fd10cb1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:51:10.358947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.355828 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46zx\" (UniqueName: \"kubernetes.io/projected/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-kube-api-access-m46zx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:10.358947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.355880 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:10.358947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.355904 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:10.364738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.364263 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 00:51:10.365442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.365420 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" containerName="revoke-keys" Oct 30 00:51:10.365499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.365445 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" containerName="revoke-keys" Oct 30 00:51:10.365634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.365619 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" containerName="revoke-keys" Oct 30 00:51:10.369403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.369384 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.372613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.372591 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 00:51:10.397749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.397705 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5f387c-openshift-default-dockercfg-7wl24\"" Oct 30 00:51:10.436344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.430366 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.459334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459305 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459397 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459421 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459483 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459571 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459620 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxm6w\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459640 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.459675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.459666 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.535704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.535567 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:10.536001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.535973 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:10.550962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.550933 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" event={"ID":"b50919b3-d60b-4b6e-94dc-7d54fd10cb1d","Type":"ContainerDied","Data":"e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7"} Oct 30 00:51:10.551476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.551463 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99fff2ff6bca100ef494464911d72372ba2ae5b8a747154321a35a49b412db7" Oct 30 00:51:10.551604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.551398 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561564 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561591 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561649 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561715 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561778 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxm6w\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561796 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561822 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.561916 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.562513 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.562537 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.562954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.562765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.563181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.563074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.563422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.563390 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.565041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.565001 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.565213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.565197 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.602885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.584220 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.602885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.584315 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxm6w\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\") pod \"e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.627906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.627497 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.757887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.751154 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:10.781479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.773020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.966890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:10.942768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:11.179060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.177314 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:11.179060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.177367 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:11.240130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.240096 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:11.406796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.406711 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:11.503795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.483544 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 00:51:11.512922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.512880 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:11.517359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.517316 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:11.517585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.517548 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:11.571367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.556234 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:11.571367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.556292 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:11.581103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:11.581056 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerStarted","Data":"8a1dad4c4b6563eeed012dee084bf4930f7064f95389c7915aa8a296f3beda91"} Oct 30 00:51:12.182334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.182296 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:12.182524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.182359 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:12.246893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.233053 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:12.546942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.543309 5938 patch_prober.go:29] interesting pod/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 00:51:12.546942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.543372 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 00:51:12.912889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.905934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerStarted","Data":"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a"} Oct 30 00:51:12.912889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:12.906524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:13.050422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.050365 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podStartSLOduration=3.050347241 podStartE2EDuration="3.050347241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:13.019285511 +0000 UTC m=+1125.259520625" watchObservedRunningTime="2026-10-30 00:51:13.050347241 +0000 UTC m=+1125.290582354" Oct 30 00:51:13.055212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.055180 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:13.057769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.057738 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg"] Oct 30 00:51:13.061855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.061829 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be862c05-km4zg"] Oct 30 00:51:13.174022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.171256 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:13.174022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.171316 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:13.273888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.267923 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:13.392428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.392388 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:51:13.392932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.392900 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" containerID="cri-o://4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1" gracePeriod=30 Oct 30 00:51:13.603890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.578954 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:13.752076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.752044 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:13.758183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.758163 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:13.878439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.878397 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3805acf5-5517-47f2-9556-793c2e7b9eed" path="/var/lib/kubelet/pods/3805acf5-5517-47f2-9556-793c2e7b9eed/volumes" Oct 30 00:51:13.914898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.913946 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:51:13.930263 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.930237 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:13.931752 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:13.931725 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:51:14.171128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:14.171057 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 00:51:14.171281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:14.171120 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 00:51:14.191028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:14.190985 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.348045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:14.348005 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.511343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:14.511263 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:15.004790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.004751 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:51:15.005279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.004996 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" containerID="cri-o://a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b" gracePeriod=30 Oct 30 00:51:15.017559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.017531 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:51:15.017763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.017736 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" containerID="cri-o://e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd" gracePeriod=30 Oct 30 00:51:15.024671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.024645 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:51:15.024937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.024911 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" containerID="cri-o://f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f" gracePeriod=30 Oct 30 00:51:15.034389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.034362 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:51:15.034597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.034572 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" containerID="cri-o://8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4" gracePeriod=30 Oct 30 00:51:15.067020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.066981 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:15.228232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.228181 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:15.228772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.228748 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:15.229477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.229456 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 00:51:15.494327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.494285 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:15.548973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.548948 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:51:15.575593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575524 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.575593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575556 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.575791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575599 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.575791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575701 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.575791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575745 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.576057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575801 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:15.576057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.576057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575932 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:15.576057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.575944 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.576057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576046 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.576343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576092 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\") pod \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\" (UID: \"788989e3-39e9-4b3c-a2e2-e9e75aed847b\") " Oct 30 00:51:15.576343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576236 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:15.576343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:15.576680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576658 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:15.576951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576927 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.576951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576950 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.576951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576959 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.577196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576966 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.577196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.576976 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.578065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.578041 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:15.578172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.578141 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:15.578357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.578335 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:15.578440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.578426 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj" pod "788989e3-39e9-4b3c-a2e2-e9e75aed847b" (UID: "788989e3-39e9-4b3c-a2e2-e9e75aed847b"). InnerVolumeSpecName "kube-api-access-7qbvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:15.650430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.650372 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:15.679190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.679131 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.679190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.679165 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.679190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.679179 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbvj\" (UniqueName: \"kubernetes.io/projected/788989e3-39e9-4b3c-a2e2-e9e75aed847b-kube-api-access-7qbvj\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.679190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.679191 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788989e3-39e9-4b3c-a2e2-e9e75aed847b-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:15.950535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.950498 5938 generic.go:356] "Generic (PLEG): container finished" podID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerID="cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5" exitCode=0 Oct 30 00:51:15.950690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.950541 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" Oct 30 00:51:15.950690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.950577 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerDied","Data":"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5"} Oct 30 00:51:15.950690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.950625 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt" event={"ID":"788989e3-39e9-4b3c-a2e2-e9e75aed847b","Type":"ContainerDied","Data":"deff7e60475613dd1be386863d5a7754319a5e2fe23178c306f46855f12de124"} Oct 30 00:51:15.950690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.950657 5938 scope.go:122] "RemoveContainer" containerID="cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5" Oct 30 00:51:15.962910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.962894 5938 scope.go:122] "RemoveContainer" containerID="b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed" Oct 30 00:51:15.975431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.975390 5938 scope.go:122] "RemoveContainer" containerID="cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5" Oct 30 00:51:15.975714 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:15.975693 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5\": container with ID starting with cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5 not found: ID does not exist" containerID="cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5" Oct 30 00:51:15.975826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.975723 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5"} err="failed to get container status \"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5\": rpc error: code = NotFound desc = could not find container \"cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5\": container with ID starting with cb9dd1033e12376f14217a197e6a53d9e25f00bdae1e9d4dc7a5c670326e49a5 not found: ID does not exist" Oct 30 00:51:15.975826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.975747 5938 scope.go:122] "RemoveContainer" containerID="b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed" Oct 30 00:51:15.976096 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:15.976078 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed\": container with ID starting with b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed not found: ID does not exist" containerID="b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed" Oct 30 00:51:15.976148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:15.976100 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed"} err="failed to get container status \"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed\": rpc error: code = NotFound desc = could not find container \"b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed\": container with ID starting with b19809186b2939bac0686c71218c2cb43ed41a43d029b8ebeb4dcefefe43b3ed not found: ID does not exist" Oct 30 00:51:16.004104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:16.004078 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:51:16.015418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:16.015396 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-751d6789-openshift-default-986c6d7-lfbwt"] Oct 30 00:51:17.097768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.097726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:17.239417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.239383 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:17.523894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.516316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:17.737972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.737928 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn"] Oct 30 00:51:17.739237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.739204 5938 cpu_manager.go:414] "containerMap: removing container" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:17.739237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.739236 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:17.739523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.739452 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:17.739598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.739540 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:17.743521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.743495 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.757831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.757122 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn"] Oct 30 00:51:17.761403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.761381 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:51:17.761546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.761448 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:51:17.761642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.761571 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:51:17.769224 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.769201 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:51:17.847069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.846992 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.847206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.847113 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.847251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.847229 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wkd\" (UniqueName: \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.879402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.879369 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" path="/var/lib/kubelet/pods/788989e3-39e9-4b3c-a2e2-e9e75aed847b/volumes" Oct 30 00:51:17.949204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.949171 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.949343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.949256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.949408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.949342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wkd\" (UniqueName: \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.949994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.949971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.951613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.951589 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:17.974571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:17.974526 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wkd\" (UniqueName: \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\") pod \"maas-api-revoke-sub-df76b0f8-5hwsn\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:18.061620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:18.061594 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:18.366640 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:18.366611 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325a495b_a3b8_45fb_8eec_dd6e66494b6f.slice/crio-a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4 WatchSource:0}: Error finding container a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4: Status 404 returned error can't find the container with id a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4 Oct 30 00:51:18.368499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:18.368472 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn"] Oct 30 00:51:18.955388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:18.955321 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:51:18.988937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:18.988901 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" event={"ID":"325a495b-a3b8-45fb-8eec-dd6e66494b6f","Type":"ContainerStarted","Data":"f8d02666d31e281a7d18d6a4de650e9e1ee55c8ca146b5cce8c7a697dbe01dc5"} Oct 30 00:51:18.989097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:18.988961 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" event={"ID":"325a495b-a3b8-45fb-8eec-dd6e66494b6f","Type":"ContainerStarted","Data":"a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4"} Oct 30 00:51:19.041666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.041619 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" podStartSLOduration=2.041607791 podStartE2EDuration="2.041607791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:19.027389914 +0000 UTC m=+1131.267625060" watchObservedRunningTime="2026-10-30 00:51:19.041607791 +0000 UTC m=+1131.281842905" Oct 30 00:51:19.064254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.064233 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:19.465131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.465110 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:51:19.580445 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580367 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580453 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580502 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580564 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580589 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580629 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580670 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:19.580843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580711 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.580843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580803 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\") pod \"6113b106-66ff-403a-a697-f62657225c91\" (UID: \"6113b106-66ff-403a-a697-f62657225c91\") " Oct 30 00:51:19.581177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580853 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:19.581177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.580905 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:19.581289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581203 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:19.581412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581392 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:19.581613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581589 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.581701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581623 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.581701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581640 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.581701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581653 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.581701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.581668 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6113b106-66ff-403a-a697-f62657225c91-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.582834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.582804 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:19.583195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.583174 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "kube-api-access-8qxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:19.583322 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.583308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:19.583435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.583415 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token" pod "6113b106-66ff-403a-a697-f62657225c91" (UID: "6113b106-66ff-403a-a697-f62657225c91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:19.682542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.682515 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6113b106-66ff-403a-a697-f62657225c91-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.682542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.682540 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6113b106-66ff-403a-a697-f62657225c91-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.682688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.682550 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.682688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.682559 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxjk\" (UniqueName: \"kubernetes.io/projected/6113b106-66ff-403a-a697-f62657225c91-kube-api-access-8qxjk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:19.999637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.999605 5938 generic.go:356] "Generic (PLEG): container finished" podID="6113b106-66ff-403a-a697-f62657225c91" containerID="4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1" exitCode=0 Oct 30 00:51:19.999772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.999654 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" Oct 30 00:51:19.999772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.999686 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerDied","Data":"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1"} Oct 30 00:51:19.999772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.999743 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m" event={"ID":"6113b106-66ff-403a-a697-f62657225c91","Type":"ContainerDied","Data":"5e23738e7e70dabe99a9d8d4c4a221300e489062143cd5e90ec19e755c68af24"} Oct 30 00:51:19.999981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:19.999772 5938 scope.go:122] "RemoveContainer" containerID="4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1" Oct 30 00:51:20.001713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.001690 5938 generic.go:356] "Generic (PLEG): container finished" podID="325a495b-a3b8-45fb-8eec-dd6e66494b6f" containerID="f8d02666d31e281a7d18d6a4de650e9e1ee55c8ca146b5cce8c7a697dbe01dc5" exitCode=0 Oct 30 00:51:20.001924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.001745 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" event={"ID":"325a495b-a3b8-45fb-8eec-dd6e66494b6f","Type":"ContainerDied","Data":"f8d02666d31e281a7d18d6a4de650e9e1ee55c8ca146b5cce8c7a697dbe01dc5"} Oct 30 00:51:20.011670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.011644 5938 scope.go:122] "RemoveContainer" containerID="105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b" Oct 30 00:51:20.022624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.022604 5938 scope.go:122] "RemoveContainer" containerID="4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1" Oct 30 00:51:20.023088 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:20.023065 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1\": container with ID starting with 4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1 not found: ID does not exist" containerID="4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1" Oct 30 00:51:20.023188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.023097 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1"} err="failed to get container status \"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1\": rpc error: code = NotFound desc = could not find container \"4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1\": container with ID starting with 4aacbbb405a9b93a470a78c3e7fe3c9be748907dfc7144afb4e66ada7aec2ee1 not found: ID does not exist" Oct 30 00:51:20.023188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.023122 5938 scope.go:122] "RemoveContainer" containerID="105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b" Oct 30 00:51:20.023437 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:20.023414 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b\": container with ID starting with 105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b not found: ID does not exist" containerID="105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b" Oct 30 00:51:20.023490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.023444 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b"} err="failed to get container status \"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b\": rpc error: code = NotFound desc = could not find container \"105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b\": container with ID starting with 105a405a275493b049001e95d5e74da5f62d695dbf63e1e930c6262b85bb5a1b not found: ID does not exist" Oct 30 00:51:20.066566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.066410 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:51:20.071100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.071073 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-be862c05-openshift-default-546c6fcf65-zvh4m"] Oct 30 00:51:20.394635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.394561 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:51:20.428161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.428141 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:51:20.432141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.432124 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:51:20.435900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.435839 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:51:20.495983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.495951 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496015 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496044 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496074 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496101 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496136 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496237 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496282 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496340 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496366 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496396 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496438 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496470 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496507 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496535 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496583 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496629 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496655 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.496702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496681 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.497024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496723 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.497024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.497024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496781 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.497024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.496922 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497008 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.497191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497040 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.497191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497083 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.497191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497248 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497305 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497334 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497387 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497432 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.497533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497487 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\") pod \"f70d7096-5107-48f2-8732-34d17da6a020\" (UID: \"f70d7096-5107-48f2-8732-34d17da6a020\") " Oct 30 00:51:20.497589 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497561 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\") pod \"e2632b1f-0282-4768-87f5-3b476146a536\" (UID: \"e2632b1f-0282-4768-87f5-3b476146a536\") " Oct 30 00:51:20.497645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497593 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\") pod \"cd4337f8-db9f-4a1b-b47a-8839844fe021\" (UID: \"cd4337f8-db9f-4a1b-b47a-8839844fe021\") " Oct 30 00:51:20.497645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497629 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\") pod \"94636f9e-16ba-4105-910f-624db998a7ca\" (UID: \"94636f9e-16ba-4105-910f-624db998a7ca\") " Oct 30 00:51:20.497752 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.497638 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498209 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498405 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498466 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498722 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:20.499028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.498745 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:20.501537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.501508 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.502272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.501973 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.502272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502059 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:20.502445 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502359 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.502511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502477 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502505 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502522 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502548 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502563 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502577 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502494 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.502659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502523 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.503003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502664 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.503003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502800 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.503003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.502838 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:20.503168 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.503046 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.503565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.503535 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.503664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.503591 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.505851 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.505981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.505926 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:20.506047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.505976 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.506099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.506036 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "kube-api-access-tjvcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.506961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.506939 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "kube-api-access-97q26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.507386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.507361 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.508260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.508235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:20.508783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.508758 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.509261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509202 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:20.509261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509213 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token" pod "94636f9e-16ba-4105-910f-624db998a7ca" (UID: "94636f9e-16ba-4105-910f-624db998a7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.509261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509217 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.509261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509225 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:20.509506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509285 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo" pod "f70d7096-5107-48f2-8732-34d17da6a020" (UID: "f70d7096-5107-48f2-8732-34d17da6a020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:51:20.509506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509304 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.509506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.509329 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln" pod "e2632b1f-0282-4768-87f5-3b476146a536" (UID: "e2632b1f-0282-4768-87f5-3b476146a536"). InnerVolumeSpecName "kube-api-access-6h5ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.510447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.510428 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl" pod "cd4337f8-db9f-4a1b-b47a-8839844fe021" (UID: "cd4337f8-db9f-4a1b-b47a-8839844fe021"). InnerVolumeSpecName "kube-api-access-xslsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:20.603826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603795 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.603826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603823 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.603826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603836 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslsl\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-kube-api-access-xslsl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603873 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603886 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70d7096-5107-48f2-8732-34d17da6a020-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603898 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603908 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603919 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603934 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5ln\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-kube-api-access-6h5ln\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603946 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603959 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2632b1f-0282-4768-87f5-3b476146a536-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603971 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603983 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94636f9e-16ba-4105-910f-624db998a7ca-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.603994 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604006 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94636f9e-16ba-4105-910f-624db998a7ca-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604017 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604030 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70d7096-5107-48f2-8732-34d17da6a020-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604043 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604058 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604073 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604085 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94636f9e-16ba-4105-910f-624db998a7ca-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604097 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2632b1f-0282-4768-87f5-3b476146a536-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604110 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4337f8-db9f-4a1b-b47a-8839844fe021-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604124 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvcz\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-kube-api-access-tjvcz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604137 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2632b1f-0282-4768-87f5-3b476146a536-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604149 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70d7096-5107-48f2-8732-34d17da6a020-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604163 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2632b1f-0282-4768-87f5-3b476146a536-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604176 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97q26\" (UniqueName: \"kubernetes.io/projected/94636f9e-16ba-4105-910f-624db998a7ca-kube-api-access-97q26\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604189 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70d7096-5107-48f2-8732-34d17da6a020-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.604827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.604204 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4337f8-db9f-4a1b-b47a-8839844fe021-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:20.700042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:20.699919 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:21.012066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.012037 5938 generic.go:356] "Generic (PLEG): container finished" podID="e2632b1f-0282-4768-87f5-3b476146a536" containerID="f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f" exitCode=0 Oct 30 00:51:21.012200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.012086 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" Oct 30 00:51:21.012200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.012122 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" event={"ID":"e2632b1f-0282-4768-87f5-3b476146a536","Type":"ContainerDied","Data":"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f"} Oct 30 00:51:21.012200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.012175 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86" event={"ID":"e2632b1f-0282-4768-87f5-3b476146a536","Type":"ContainerDied","Data":"df3d7094db9ec5f73a71ca0a5187a3bead90d38203015d6f4b4b6d26516ed4d0"} Oct 30 00:51:21.012200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.012198 5938 scope.go:122] "RemoveContainer" containerID="f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f" Oct 30 00:51:21.014209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.014183 5938 generic.go:356] "Generic (PLEG): container finished" podID="94636f9e-16ba-4105-910f-624db998a7ca" containerID="8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4" exitCode=0 Oct 30 00:51:21.014346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.014237 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" Oct 30 00:51:21.014346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.014258 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" event={"ID":"94636f9e-16ba-4105-910f-624db998a7ca","Type":"ContainerDied","Data":"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4"} Oct 30 00:51:21.014346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.014312 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g" event={"ID":"94636f9e-16ba-4105-910f-624db998a7ca","Type":"ContainerDied","Data":"028d807ab238d202195a9cb2ff415df4512b72d95301394da90b72c4cda9b5a1"} Oct 30 00:51:21.016161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.016134 5938 generic.go:356] "Generic (PLEG): container finished" podID="f70d7096-5107-48f2-8732-34d17da6a020" containerID="a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b" exitCode=0 Oct 30 00:51:21.016298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.016170 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" Oct 30 00:51:21.016298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.016208 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" event={"ID":"f70d7096-5107-48f2-8732-34d17da6a020","Type":"ContainerDied","Data":"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b"} Oct 30 00:51:21.016298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.016254 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn" event={"ID":"f70d7096-5107-48f2-8732-34d17da6a020","Type":"ContainerDied","Data":"6fc1d8d347cf0519d77b652e54f61676b3c7b0e9c419a71c057bf46f0786df36"} Oct 30 00:51:21.024760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.024737 5938 generic.go:356] "Generic (PLEG): container finished" podID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerID="e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd" exitCode=0 Oct 30 00:51:21.024853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.024777 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" Oct 30 00:51:21.024853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.024805 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" event={"ID":"cd4337f8-db9f-4a1b-b47a-8839844fe021","Type":"ContainerDied","Data":"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd"} Oct 30 00:51:21.024853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.024839 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp" event={"ID":"cd4337f8-db9f-4a1b-b47a-8839844fe021","Type":"ContainerDied","Data":"0cc651565da0ea1e941c0c60f441f556876816db01428549376d84d0f7e380c2"} Oct 30 00:51:21.031346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.031257 5938 scope.go:122] "RemoveContainer" containerID="f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f" Oct 30 00:51:21.032099 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:21.031834 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f\": container with ID starting with f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f not found: ID does not exist" containerID="f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f" Oct 30 00:51:21.032099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.031895 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f"} err="failed to get container status \"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f\": rpc error: code = NotFound desc = could not find container \"f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f\": container with ID starting with f3b426f3719058f4a6b5d87d4c4dc02b54b0bad452c5e9e58dd7919c599e2c3f not found: ID does not exist" Oct 30 00:51:21.032099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.031919 5938 scope.go:122] "RemoveContainer" containerID="8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4" Oct 30 00:51:21.043635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.043617 5938 scope.go:122] "RemoveContainer" containerID="8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4" Oct 30 00:51:21.043913 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:21.043894 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4\": container with ID starting with 8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4 not found: ID does not exist" containerID="8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4" Oct 30 00:51:21.044002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.043917 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4"} err="failed to get container status \"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4\": rpc error: code = NotFound desc = could not find container \"8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4\": container with ID starting with 8d697074709d52f7e04c9e9f08a085b2fbcc2ee4158ba1eb3f719985fcb72aa4 not found: ID does not exist" Oct 30 00:51:21.044002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.043938 5938 scope.go:122] "RemoveContainer" containerID="a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b" Oct 30 00:51:21.054149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.054131 5938 scope.go:122] "RemoveContainer" containerID="a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b" Oct 30 00:51:21.054443 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:21.054428 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b\": container with ID starting with a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b not found: ID does not exist" containerID="a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b" Oct 30 00:51:21.054504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.054450 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b"} err="failed to get container status \"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b\": rpc error: code = NotFound desc = could not find container \"a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b\": container with ID starting with a3a86e802a825538870cd59338d4472e1d04f755958f42070f7d5ba1f5996d5b not found: ID does not exist" Oct 30 00:51:21.054504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.054467 5938 scope.go:122] "RemoveContainer" containerID="e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd" Oct 30 00:51:21.067132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.067114 5938 scope.go:122] "RemoveContainer" containerID="e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd" Oct 30 00:51:21.067442 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:21.067417 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd\": container with ID starting with e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd not found: ID does not exist" containerID="e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd" Oct 30 00:51:21.067524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.067452 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd"} err="failed to get container status \"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd\": rpc error: code = NotFound desc = could not find container \"e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd\": container with ID starting with e23b5eda1d15fd06eb30221029951943eccc404b59cf96f0cf319a6cf70a53cd not found: ID does not exist" Oct 30 00:51:21.130621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.130558 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:51:21.138649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.138620 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-s47sp"] Oct 30 00:51:21.354850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.354819 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:51:21.363010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.362341 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ws84g"] Oct 30 00:51:21.368333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.368305 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:51:21.372668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.372649 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lxc86"] Oct 30 00:51:21.377692 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.377672 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:51:21.388485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.388463 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-gfvgn"] Oct 30 00:51:21.401005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.400978 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:21.516624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.516590 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\" (UniqueName: \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\") pod \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " Oct 30 00:51:21.517051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.516660 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\") pod \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " Oct 30 00:51:21.517051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.516945 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\") pod \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\" (UID: \"325a495b-a3b8-45fb-8eec-dd6e66494b6f\") " Oct 30 00:51:21.517051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.517040 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca" pod "325a495b-a3b8-45fb-8eec-dd6e66494b6f" (UID: "325a495b-a3b8-45fb-8eec-dd6e66494b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:51:21.517636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.517614 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:21.518774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.518750 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd" pod "325a495b-a3b8-45fb-8eec-dd6e66494b6f" (UID: "325a495b-a3b8-45fb-8eec-dd6e66494b6f"). InnerVolumeSpecName "kube-api-access-v8wkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:21.518924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.518905 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca" pod "325a495b-a3b8-45fb-8eec-dd6e66494b6f" (UID: "325a495b-a3b8-45fb-8eec-dd6e66494b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:51:21.619262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.619180 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325a495b-a3b8-45fb-8eec-dd6e66494b6f-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:21.619262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.619211 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wkd\" (UniqueName: \"kubernetes.io/projected/325a495b-a3b8-45fb-8eec-dd6e66494b6f-kube-api-access-v8wkd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:51:21.879316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.879238 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6113b106-66ff-403a-a697-f62657225c91" path="/var/lib/kubelet/pods/6113b106-66ff-403a-a697-f62657225c91/volumes" Oct 30 00:51:21.879812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.879797 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94636f9e-16ba-4105-910f-624db998a7ca" path="/var/lib/kubelet/pods/94636f9e-16ba-4105-910f-624db998a7ca/volumes" Oct 30 00:51:21.880335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.880320 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" path="/var/lib/kubelet/pods/cd4337f8-db9f-4a1b-b47a-8839844fe021/volumes" Oct 30 00:51:21.881145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.881129 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2632b1f-0282-4768-87f5-3b476146a536" path="/var/lib/kubelet/pods/e2632b1f-0282-4768-87f5-3b476146a536/volumes" Oct 30 00:51:21.881591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:21.881577 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70d7096-5107-48f2-8732-34d17da6a020" path="/var/lib/kubelet/pods/f70d7096-5107-48f2-8732-34d17da6a020/volumes" Oct 30 00:51:22.121464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:22.119028 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" event={"ID":"325a495b-a3b8-45fb-8eec-dd6e66494b6f","Type":"ContainerDied","Data":"a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4"} Oct 30 00:51:22.121464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:22.119092 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7beb742caaf7b19177167b8ae596bce373c5cf45bc588084fb16ee597247fa4" Oct 30 00:51:22.121464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:22.119181 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn" Oct 30 00:51:22.509063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:22.507967 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:23.193888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.193167 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:23.672945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.672909 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:51:23.952969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.734177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:23.991029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.990989 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" Oct 30 00:51:23.991246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.991227 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" Oct 30 00:51:23.994889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.991384 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:24.004892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:23.995051 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:24.005102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.005080 5938 cpu_manager.go:414] "containerMap: removing container" podUID="325a495b-a3b8-45fb-8eec-dd6e66494b6f" containerName="revoke-keys" Oct 30 00:51:24.005255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.005240 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325a495b-a3b8-45fb-8eec-dd6e66494b6f" containerName="revoke-keys" Oct 30 00:51:24.012888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.005405 5938 cpu_manager.go:414] "containerMap: removing container" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" Oct 30 00:51:24.013118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.013096 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" Oct 30 00:51:24.013297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.013271 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" Oct 30 00:51:24.013396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.013385 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" Oct 30 00:51:24.049902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.013491 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" Oct 30 00:51:24.095455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.095411 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" Oct 30 00:51:24.101966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.095726 5938 cpu_manager.go:414] "containerMap: removing container" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:24.102177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.102160 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788989e3-39e9-4b3c-a2e2-e9e75aed847b" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109069 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="94636f9e-16ba-4105-910f-624db998a7ca" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109118 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2632b1f-0282-4768-87f5-3b476146a536" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109129 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70d7096-5107-48f2-8732-34d17da6a020" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109155 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4337f8-db9f-4a1b-b47a-8839844fe021" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109164 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109188 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="325a495b-a3b8-45fb-8eec-dd6e66494b6f" containerName="revoke-keys" Oct 30 00:51:24.127887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.109201 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:24.160887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.157983 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.224154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.214184 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d7a6db25-openshift-default-dockercfg-qnm52\"" Oct 30 00:51:24.224154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.214669 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:24.251249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.250704 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.251249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.250827 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.251249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.250902 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290102 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290253 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290323 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290376 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290468 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.291890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.290607 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpvm\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.433957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.433919 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:51:24.496122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.494940 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.499407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.499378 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504134 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504274 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504390 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504423 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.504576 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpvm\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.500735 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.507884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.505310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.535884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.509228 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.536397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.523148 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.536550 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.523726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.536901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.536848 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.571284 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.568181 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.571284 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.568398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpvm\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.579005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.577335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.615885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.612512 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.675251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.675218 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:24.682908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.675921 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.682908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.677184 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 00:51:24.723139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.713645 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:24.865887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.857504 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:24.886887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.879948 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:24.886887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.880280 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 00:51:24.924835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:24.921519 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:51:25.071080 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.070975 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.148006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.147960 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.173082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.148081 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.250364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.148121 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:51:25.426506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.426042 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:51:25.427160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.427135 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:25.439231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.439210 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:51:25.460038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.460006 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:25.460153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.460085 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:25.460153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:25.460143 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:26.381645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.381605 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerStarted","Data":"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a"} Oct 30 00:51:26.382040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.381673 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerStarted","Data":"ed260754d17c7d17f24a29eb5b50d84807a9e1a58478323f3e559fb7ca01d51a"} Oct 30 00:51:26.431799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.431766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:26.466029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.465978 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podStartSLOduration=3.46596079 podStartE2EDuration="3.46596079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:26.438138276 +0000 UTC m=+1138.678373402" watchObservedRunningTime="2026-10-30 00:51:26.46596079 +0000 UTC m=+1138.706195900" Oct 30 00:51:26.675781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.675684 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:26.693890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.682990 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:26.935969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:26.935855 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:27.074886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:27.074828 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:27.416442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:27.415117 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:27.416442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:27.416214 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:51:28.130693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.130647 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:28.407941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.407360 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:28.781135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.781067 5938 scope.go:122] "RemoveContainer" containerID="44046fb0887e202c4280c6e9c8cc1e0cf5e10338de948f2dfdbe97888e3868c8" Oct 30 00:51:28.793514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.793306 5938 scope.go:122] "RemoveContainer" containerID="afd175218fe02c946b1dd01809859c18c632e67067e87ea9fa43ce694d56178f" Oct 30 00:51:28.803938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.803921 5938 scope.go:122] "RemoveContainer" containerID="8881b687cd943f8dac23c84399e1af5b3bb0c2e164f839936d8b970263eff5c2" Oct 30 00:51:28.813038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.813020 5938 scope.go:122] "RemoveContainer" containerID="af5ad5a469341653b68484b2ec89ecfca073e5f2a5c4edab7a44ff967f73c430" Oct 30 00:51:28.823942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.823844 5938 scope.go:122] "RemoveContainer" containerID="cb6022be04b6d4307608aed9b98eaebb9d03a4bb43f4320bbc3b7064cbaf2707" Oct 30 00:51:28.834094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.834077 5938 scope.go:122] "RemoveContainer" containerID="55ce76580a892001d53d284101b83f072fc5e727e7b609a7b1410e940968d81c" Oct 30 00:51:28.844782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.844762 5938 scope.go:122] "RemoveContainer" containerID="db29c376d356ad89eb9b29262b7d239c0319ff1b4dd7e4f6b892d761347bf24a" Oct 30 00:51:28.845250 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:28.845219 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:51:28.854095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.854074 5938 scope.go:122] "RemoveContainer" containerID="fd6d885ef2db48152f88cc1a090c983177c921914591f710cd72cd1812de6d70" Oct 30 00:51:28.863496 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.863480 5938 scope.go:122] "RemoveContainer" containerID="1998abb9f44bf879de0c33fcdb687775540f9aa58dd8846213ec45750dc4adf3" Oct 30 00:51:28.872415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.872398 5938 scope.go:122] "RemoveContainer" containerID="5558a968d71c74b6448ae5cd45e64c1299c06846da03a45bbe951066cc376f72" Oct 30 00:51:28.881020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.881003 5938 scope.go:122] "RemoveContainer" containerID="4e5c67b0b2e6f9cc2dbffae65cbb7614592712e95e91f61c03ee155d91c3ecc3" Oct 30 00:51:28.956825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:28.956798 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:29.106466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:29.106436 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:30.376217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:30.376179 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.007273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.007237 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.153992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.153939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.329780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.329686 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:51:32.337128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.331655 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.337128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.332200 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:32.337128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.332221 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6113b106-66ff-403a-a697-f62657225c91" containerName="istio-proxy" Oct 30 00:51:32.361395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.361361 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.393516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.393455 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d7a6db25-dockercfg-7285h\"" Oct 30 00:51:32.410064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.408299 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:51:32.426553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.423200 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d7a6db25\"" Oct 30 00:51:32.529347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.529313 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.529469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.529379 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcm9\" (UniqueName: \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.630728 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.630693 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.630949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.630747 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcm9\" (UniqueName: \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.631445 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.631425 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.653336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.653310 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcm9\" (UniqueName: \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:32.761151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:32.761127 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:33.029207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.029121 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4"] Oct 30 00:51:33.040645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.040616 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b9bc9e-grrn4"] Oct 30 00:51:33.091079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.091049 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:51:33.091198 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:33.091151 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8f658f_e268_474b_87dd_2a0576f7c577.slice/crio-45e52f1e6232de84d4f7d4eb542bdda9fa4aff7b85a9e416b66b3c72a0b0fb77 WatchSource:0}: Error finding container 45e52f1e6232de84d4f7d4eb542bdda9fa4aff7b85a9e416b66b3c72a0b0fb77: Status 404 returned error can't find the container with id 45e52f1e6232de84d4f7d4eb542bdda9fa4aff7b85a9e416b66b3c72a0b0fb77 Oct 30 00:51:33.517511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.517473 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" event={"ID":"6d8f658f-e268-474b-87dd-2a0576f7c577","Type":"ContainerStarted","Data":"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643"} Oct 30 00:51:33.517720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.517536 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" event={"ID":"6d8f658f-e268-474b-87dd-2a0576f7c577","Type":"ContainerStarted","Data":"45e52f1e6232de84d4f7d4eb542bdda9fa4aff7b85a9e416b66b3c72a0b0fb77"} Oct 30 00:51:33.517720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.517687 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:33.520313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.520285 5938 generic.go:356] "Generic (PLEG): container finished" podID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerID="0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5" exitCode=0 Oct 30 00:51:33.520456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.520349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerDied","Data":"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5"} Oct 30 00:51:33.520828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.520813 5938 scope.go:122] "RemoveContainer" containerID="0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5" Oct 30 00:51:33.522333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.522306 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:33.522412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.522378 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:33.522449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.522413 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:33.563646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.563603 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" podStartSLOduration=1.563592493 podStartE2EDuration="1.563592493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:33.560687392 +0000 UTC m=+1145.800922507" watchObservedRunningTime="2026-10-30 00:51:33.563592493 +0000 UTC m=+1145.803827612" Oct 30 00:51:33.794539 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.794452 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:33.879314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:33.879279 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a95512-5fd7-4f30-9ef0-b495ed6a437c" path="/var/lib/kubelet/pods/c4a95512-5fd7-4f30-9ef0-b495ed6a437c/volumes" Oct 30 00:51:34.378719 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.378686 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:34.531763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.531728 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerStarted","Data":"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce"} Oct 30 00:51:34.712780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.712215 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:51:34.712780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.712351 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:51:34.712780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.712417 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 00:51:34.752114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.752073 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:34.905889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:34.897998 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:35.221781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.221750 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 00:51:35.226002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.225979 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.251256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.240470 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8e0f66-openshift-default-dockercfg-2lhs9\"" Oct 30 00:51:35.259457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.259434 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 00:51:35.280240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280210 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280253 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280472 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbq8\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280575 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280677 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280757 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280735 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.280970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.280951 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.281048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.281005 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.320616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.320583 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:35.408910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.408749 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbq8\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.408910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.408854 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422458 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422566 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422636 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422669 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422843 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.422993 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.423027 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.423477 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.423724 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.424039 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.426882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.424657 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.460993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.458271 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.483106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.481549 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbq8\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.483106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.482013 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.509886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.508463 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\") pod \"e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.538889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.538531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:35.552889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.552121 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:35.723342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.723296 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:51:35.723540 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.723385 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 00:51:35.809889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:35.803773 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:36.089940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.087209 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 00:51:36.111343 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:36.111301 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9453dd2_521d_4380_ba5d_b2208bd3dccf.slice/crio-99c0bdf7968f853d67cea6eb37855f860d636f3818753399bb40d52bcc3ae125 WatchSource:0}: Error finding container 99c0bdf7968f853d67cea6eb37855f860d636f3818753399bb40d52bcc3ae125: Status 404 returned error can't find the container with id 99c0bdf7968f853d67cea6eb37855f860d636f3818753399bb40d52bcc3ae125 Oct 30 00:51:36.114164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.114131 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:36.114256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.114195 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:36.114256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.114227 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:36.364588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.364552 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:36.642512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.642425 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerStarted","Data":"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d"} Oct 30 00:51:36.642512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.642482 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerStarted","Data":"99c0bdf7968f853d67cea6eb37855f860d636f3818753399bb40d52bcc3ae125"} Oct 30 00:51:36.706910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.702623 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:51:36.706910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.702676 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 00:51:36.720998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:36.720940 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podStartSLOduration=1.72092564 podStartE2EDuration="1.72092564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:36.702538756 +0000 UTC m=+1148.942773870" watchObservedRunningTime="2026-10-30 00:51:36.72092564 +0000 UTC m=+1148.961160752" Oct 30 00:51:37.422135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.422107 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:51:37.460732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.460702 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:37.552784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.552740 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:37.559556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.559526 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:37.651005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.650976 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:37.652030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.652008 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:51:37.707938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.707544 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:51:37.707938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.707597 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 00:51:37.804820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:37.803725 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:38.705889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:38.704145 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 00:51:38.705889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:38.704201 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 00:51:38.850350 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:38.850312 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:39.148556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:39.148519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:39.311369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:39.311315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:39.794945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:39.794887 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:51:39.796665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:39.796634 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 00:51:40.397531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:40.397500 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:40.824712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:40.824630 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:41.057884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:41.057800 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:41.228693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:41.228662 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:42.764099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:42.764059 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:44.274691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:44.274656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:44.535280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:44.535206 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:51:44.942849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:44.942816 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:45.960061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:45.960026 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:46.098609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:46.098572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:46.280635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:46.280547 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:47.661279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:47.661248 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:49.331747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:49.331667 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:50.046794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:50.046753 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5"] Oct 30 00:51:50.056578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:50.056550 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3c2dcc-24cd5"] Oct 30 00:51:50.933265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:50.933235 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:51.887848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:51.887801 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b04b3ff-1670-47b5-aff4-3913b1a5f5e5" path="/var/lib/kubelet/pods/5b04b3ff-1670-47b5-aff4-3913b1a5f5e5/volumes" Oct 30 00:51:52.897746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:52.895949 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:53.074894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.063951 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:53.074894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.064315 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded" Oct 30 00:51:53.150071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.149952 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:53.151287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.151258 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:51:53.151884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.151842 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:51:53.391208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.391146 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 00:51:53.416854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.416689 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:53.452897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.447937 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:51:53.825124 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:53.825041 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:55.932280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:55.932242 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:56.182815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:56.182726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:56.640597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:56.640566 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:57.023819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.023739 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 00:51:57.029916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.029854 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.048166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.048122 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-00e07a-openshift-default-dockercfg-jjhbl\"" Oct 30 00:51:57.067167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.067140 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 00:51:57.114828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.114786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:57.115451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115427 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.115591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115463 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.115591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115546 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.115591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115579 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.115769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115694 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.115968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.115944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.116064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.116008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.116142 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.116060 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7q2g\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.116201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.116177 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239075 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239154 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239209 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239379 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239432 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7q2g\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239601 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.239692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.240535 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.245943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.241779 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.247909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.247880 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.248800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.248780 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.252814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.252154 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.268718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.267591 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.304931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.299238 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.305637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.305608 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.306439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.306415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7q2g\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\") pod \"e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.355888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.352718 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:57.635184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.635149 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:57.814330 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:51:57.814297 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6982266f_1fbe_485d_94bf_a8267348734b.slice/crio-281556b3c497469afd2873cbb84607366773507d84382cd42b3764aa86faeca1 WatchSource:0}: Error finding container 281556b3c497469afd2873cbb84607366773507d84382cd42b3764aa86faeca1: Status 404 returned error can't find the container with id 281556b3c497469afd2873cbb84607366773507d84382cd42b3764aa86faeca1 Oct 30 00:51:57.815585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.815553 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 00:51:57.817138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.817101 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:57.817250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.817178 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:57.817250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.817229 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:51:57.823571 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.823550 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:57.904629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:57.884537 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerStarted","Data":"281556b3c497469afd2873cbb84607366773507d84382cd42b3764aa86faeca1"} Oct 30 00:51:58.713892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:58.713844 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:58.835668 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:51:58.835634 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:51:58.887760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:58.887723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerStarted","Data":"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3"} Oct 30 00:51:58.947668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:58.947619 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podStartSLOduration=2.947606223 podStartE2EDuration="2.947606223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:58.941893871 +0000 UTC m=+1171.182128986" watchObservedRunningTime="2026-10-30 00:51:58.947606223 +0000 UTC m=+1171.187841335" Oct 30 00:51:59.353750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.353715 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:59.359141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.359116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:59.591885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.591804 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:59.792237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.792118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:59.896369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.896305 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:59.897512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.897483 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:51:59.973329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.972022 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:51:59.973329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:51:59.972107 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:01.552852 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:01.552811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:01.715094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:01.715043 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:02.948044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.948009 5938 generic.go:356] "Generic (PLEG): container finished" podID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerID="b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a" exitCode=0 Oct 30 00:52:02.948465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.948082 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerDied","Data":"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a"} Oct 30 00:52:02.948465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.948397 5938 scope.go:122] "RemoveContainer" containerID="b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a" Oct 30 00:52:02.949785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.949756 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:02.949843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.949827 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:02.949909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:02.949894 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:03.268024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.267949 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:03.397678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.397644 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 00:52:03.401413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.401388 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.403921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.403876 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:03.416819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.416800 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-00e07a-dockercfg-t485z\"" Oct 30 00:52:03.416993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.416847 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-00e07a\"" Oct 30 00:52:03.421126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.421105 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 00:52:03.589937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.586032 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.589937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.586254 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8lz\" (UniqueName: \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.647856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.638443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:03.660937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.660900 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 00:52:03.666432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.666408 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.676690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.676613 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 00:52:03.698887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.695555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8lz\" (UniqueName: \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.698887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.695829 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.762375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.755979 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8lz\" (UniqueName: \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.762375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.760374 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:03.797261 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.797232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zmn\" (UniqueName: \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.797438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.797353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.899476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.899440 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zmn\" (UniqueName: \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.899690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.899543 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.900330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.900307 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.923131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.923095 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zmn\" (UniqueName: \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\") pod \"payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:03.958598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:03.958564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerStarted","Data":"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa"} Oct 30 00:52:04.017537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.017509 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:04.064359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.064327 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:04.323481 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:52:04.323449 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dce2374_c3af_4dba_9918_0f87a6d9a5d3.slice/crio-4a89e4d576bc96481915c4f6232cfc75f13e8c63da7b975248229cc0708365e1 WatchSource:0}: Error finding container 4a89e4d576bc96481915c4f6232cfc75f13e8c63da7b975248229cc0708365e1: Status 404 returned error can't find the container with id 4a89e4d576bc96481915c4f6232cfc75f13e8c63da7b975248229cc0708365e1 Oct 30 00:52:04.323918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.323890 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 00:52:04.383438 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:52:04.383406 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d38970_bce0_4aff_ba74_fe9a8e23338e.slice/crio-b2ed316f05d4a468f2bce719aa446e8a9408be4902caa008b360fd224fdd150f WatchSource:0}: Error finding container b2ed316f05d4a468f2bce719aa446e8a9408be4902caa008b360fd224fdd150f: Status 404 returned error can't find the container with id b2ed316f05d4a468f2bce719aa446e8a9408be4902caa008b360fd224fdd150f Oct 30 00:52:04.409125 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.406533 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 00:52:04.752072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.752039 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:52:04.752927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.752858 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:04.753063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.752929 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:04.968798 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.968760 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" event={"ID":"10d38970-bce0-4aff-ba74-fe9a8e23338e","Type":"ContainerStarted","Data":"30a5907c8a65448e1d7602eace65865c7bc5a95c1cdab7ebc5ed00d385c3d7d2"} Oct 30 00:52:04.969229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.969012 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:04.969229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.969053 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" event={"ID":"10d38970-bce0-4aff-ba74-fe9a8e23338e","Type":"ContainerStarted","Data":"b2ed316f05d4a468f2bce719aa446e8a9408be4902caa008b360fd224fdd150f"} Oct 30 00:52:04.970641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.970614 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" event={"ID":"9dce2374-c3af-4dba-9918-0f87a6d9a5d3","Type":"ContainerStarted","Data":"c501150713b53ff6b30e2adb47f15fe61ea1c36bef212e13f66ffddd4e07d6bc"} Oct 30 00:52:04.970736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:04.970660 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" event={"ID":"9dce2374-c3af-4dba-9918-0f87a6d9a5d3","Type":"ContainerStarted","Data":"4a89e4d576bc96481915c4f6232cfc75f13e8c63da7b975248229cc0708365e1"} Oct 30 00:52:05.009945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.009804 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" podStartSLOduration=2.009788147 podStartE2EDuration="2.009788147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:05.009193181 +0000 UTC m=+1177.249428297" watchObservedRunningTime="2026-10-30 00:52:05.009788147 +0000 UTC m=+1177.250023261" Oct 30 00:52:05.052118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.052071 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" podStartSLOduration=2.052057328 podStartE2EDuration="2.052057328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:05.047600598 +0000 UTC m=+1177.287835710" watchObservedRunningTime="2026-10-30 00:52:05.052057328 +0000 UTC m=+1177.292292484" Oct 30 00:52:05.562300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.562269 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:05.752252 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.752216 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:05.752428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.752273 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:05.978709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:05.978674 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:06.751771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:06.751735 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:06.751998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:06.751786 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:07.030037 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.029963 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h"] Oct 30 00:52:07.041801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.041778 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfba6cfd-lb94h"] Oct 30 00:52:07.134973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.134942 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:07.752029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.751994 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:07.752243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.752052 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:07.877554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:07.877523 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b42a07-b334-4aee-a379-aaaccce18c94" path="/var/lib/kubelet/pods/b5b42a07-b334-4aee-a379-aaaccce18c94/volumes" Oct 30 00:52:08.667876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:08.667835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:08.753887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:08.753283 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:08.753887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:08.753359 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:09.752603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:09.752565 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 00:52:09.753012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:09.752653 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 00:52:10.466461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:10.466430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:10.756337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:10.756268 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:52:10.756776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:10.756751 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:52:10.757462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:10.757445 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 00:52:12.380305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:12.380255 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:14.032841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:14.032804 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:14.844908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:14.844761 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:15.037889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.033691 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 00:52:15.043247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.043182 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.047096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.046206 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 00:52:15.052180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.052156 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.053096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.053075 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 00:52:15.058442 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.058421 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.061325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.061302 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 00:52:15.068265 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.068241 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwncz\"" Oct 30 00:52:15.069493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.069467 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 00:52:15.074587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.074569 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 00:52:15.150116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.150373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150339 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.150491 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150475 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9t5\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.150614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150599 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.150733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.150854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.150840 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.151046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151031 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.151171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151157 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.151275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151263 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.151403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151389 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.151519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151506 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.151624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151609 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.151729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151715 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfbr\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.151904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.151852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.152051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.152036 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.152151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.152138 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.152259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.152245 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.163248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.163222 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.163527 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.163506 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.163716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.163695 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.163940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.163920 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.164097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.164079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.169100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.169078 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.169211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.169136 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx82\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.169211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.169189 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.169331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.169307 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.169475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.169399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271164 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx82\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271214 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271239 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271266 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271313 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271333 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9t5\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271373 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271401 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271426 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271457 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271486 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271506 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271557 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271586 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271617 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271638 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfbr\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271676 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271710 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271727 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271747 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271773 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271794 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.271914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271881 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.272172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271948 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.272172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.271978 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.272172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.272006 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.272428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.272406 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.272778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.272762 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.273316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.273287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.273316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.273296 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.273549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.273526 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.274045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.274018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.274303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.274278 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.274521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.274495 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.274708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.274533 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.274823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.274803 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.275102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275077 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.275218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275195 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.275457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275432 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.275619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275588 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.275740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275716 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.275820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.275916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.275892 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.276156 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.276131 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.276355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.276338 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.276771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.276752 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.277882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.277841 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.313099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.312770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx82\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.343498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.343437 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.344175 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.344126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-fv487\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.345610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.345576 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9t5\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.348309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.348287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kxdph\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.351644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.351518 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfbr\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\") pod \"maas-default-gateway-openshift-default-7d9897b54c-jdlt8\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.371405 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.371387 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:15.381967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.381949 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:15.473066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.472557 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:15.568129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:15.568055 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:16.060639 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:52:16.060287 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb92cf4_01e8_4548_b314_115832aa7b20.slice/crio-7f59bf79e029726bdb11331f5eac2d6e1b42fa6fc5e7fb3d1a9bbe00579f44a9 WatchSource:0}: Error finding container 7f59bf79e029726bdb11331f5eac2d6e1b42fa6fc5e7fb3d1a9bbe00579f44a9: Status 404 returned error can't find the container with id 7f59bf79e029726bdb11331f5eac2d6e1b42fa6fc5e7fb3d1a9bbe00579f44a9 Oct 30 00:52:16.060639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.060325 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 00:52:16.064016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.063978 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.064133 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.064065 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.064133 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.064110 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.066556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.066527 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.066729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.066709 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.066899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.066879 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.070273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.070205 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 00:52:16.079740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.079706 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 00:52:16.179302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.178525 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 00:52:16.256183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.252427 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.256183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.252522 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.256183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.252563 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:16.274624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.274587 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" event={"ID":"bfb92cf4-01e8-4548-b314-115832aa7b20","Type":"ContainerStarted","Data":"7f59bf79e029726bdb11331f5eac2d6e1b42fa6fc5e7fb3d1a9bbe00579f44a9"} Oct 30 00:52:16.276534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.276504 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" event={"ID":"4e630456-41c9-408e-b99c-89cd312e9b78","Type":"ContainerStarted","Data":"0e2d608ad88af5a5c2697ca16b0cc3191717c68bd3ff92e4b45c3db1318b3a03"} Oct 30 00:52:16.279186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:16.279156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" event={"ID":"4b9f66cf-00e0-47ee-9431-8763de57c1b4","Type":"ContainerStarted","Data":"7fb2b880ed05e784b5fa24b0c77b4491d509b49801e0b660daa817b4a9e0b9ef"} Oct 30 00:52:17.034888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.034052 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 00:52:17.378679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.378638 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:17.516886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.516471 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" event={"ID":"bfb92cf4-01e8-4548-b314-115832aa7b20","Type":"ContainerStarted","Data":"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c"} Oct 30 00:52:17.701005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.617611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:17.871886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.778310 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" event={"ID":"4e630456-41c9-408e-b99c-89cd312e9b78","Type":"ContainerStarted","Data":"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318"} Oct 30 00:52:17.871886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.844258 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podStartSLOduration=3.844238054 podStartE2EDuration="3.844238054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:17.745353226 +0000 UTC m=+1189.985588342" watchObservedRunningTime="2026-10-30 00:52:17.844238054 +0000 UTC m=+1190.084473168" Oct 30 00:52:17.881088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:17.879924 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podStartSLOduration=3.879903653 podStartE2EDuration="3.879903653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:17.839815481 +0000 UTC m=+1190.080050595" watchObservedRunningTime="2026-10-30 00:52:17.879903653 +0000 UTC m=+1190.120138768" Oct 30 00:52:18.138117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.138067 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.141735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.141700 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.155654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.155620 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.173678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.173641 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" event={"ID":"4b9f66cf-00e0-47ee-9431-8763de57c1b4","Type":"ContainerStarted","Data":"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0"} Oct 30 00:52:18.414459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.413546 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:18.414459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.413677 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.463240 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.463320 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.464485 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.464537 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.470273 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.470616 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.545888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.517286 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.553895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.519098 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:18.553895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.525054 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:18.553895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.550072 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:18.676043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.675941 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.683102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.683060 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:18.897082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:18.897022 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:52:19.004992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.000723 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:19.075001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.074955 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.142888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.122375 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podStartSLOduration=5.122351349 podStartE2EDuration="5.122351349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:18.228953758 +0000 UTC m=+1190.469188872" watchObservedRunningTime="2026-10-30 00:52:19.122351349 +0000 UTC m=+1191.362586466" Oct 30 00:52:19.143969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.143931 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth"] Oct 30 00:52:19.155842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.155772 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.156207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.156183 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:52:19.182920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.162180 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:52:19.182920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.162242 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.392602 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.392662 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.416964 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.417521 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.417561 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.430514 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11b081af-n5hth"] Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.511471 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Oct 30 00:52:19.515351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.513493 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:52:19.573359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.569381 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.573359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.569813 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:52:19.589026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.588998 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:19.593889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.589228 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:19.682189 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.681033 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.714046 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.714007 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:52:19.742095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.736090 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:52:19.772636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.772608 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 00:52:19.773235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.773213 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:52:19.884086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.884050 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:19.968115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:19.966991 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:52:20.071159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.071128 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a21b2a-d62f-42c4-b64d-4d1fe2e7a682" path="/var/lib/kubelet/pods/03a21b2a-d62f-42c4-b64d-4d1fe2e7a682/volumes" Oct 30 00:52:20.072223 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.072189 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:52:20.101888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.075773 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:20.101888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.076293 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:52:20.101888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.077893 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 00:52:20.170888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.166024 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:20.226938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.171965 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:20.292889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.287594 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" Oct 30 00:52:20.307885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.302109 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 00:52:20.307885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.302174 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:52:20.410891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.399347 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 00:52:20.457335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.457304 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.475796 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.475856 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.479655 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.479701 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.479772 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:20.525959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.479802 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:20.573894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.571238 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:20.573894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.573086 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:20.900952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:20.854432 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:52:21.081994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.081487 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:52:21.404884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.377825 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:21.404884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.377897 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:21.469900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.467042 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:21.469900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.467104 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:21.476096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.474538 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:21.476096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:21.474590 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:22.329506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.329411 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:22.378747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.378250 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:22.378747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.378320 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:22.389155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.389026 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:22.389155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.389085 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:22.470900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.469465 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:22.488464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.488434 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:22.488745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.488716 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:22.714894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:22.714790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:23.049943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.046168 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79"] Oct 30 00:52:23.124039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.102429 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd1e492a-9mq79"] Oct 30 00:52:23.373044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.372618 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:23.373044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.372675 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:23.407917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.402476 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:23.407917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.402535 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:23.477884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.477585 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:23.477884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.477645 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:23.692953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.692033 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:23.692953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.692099 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:24.013937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:23.998585 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ce8980-713b-482c-8717-946007219012" path="/var/lib/kubelet/pods/50ce8980-713b-482c-8717-946007219012/volumes" Oct 30 00:52:24.246309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.242095 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:24.383896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.375006 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:24.383896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.375067 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:24.385495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.385376 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:24.385495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.385432 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:24.475456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.473912 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:24.475456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:24.473983 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:25.154850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.154735 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.374010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.373977 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:25.374289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.374259 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:25.383181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.383074 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:25.383181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.383117 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:25.473625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.473536 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:25.473625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:25.473598 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:26.010014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.009970 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.372008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.371963 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-fv487 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 00:52:26.372233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.372026 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 00:52:26.382706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.382670 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-jdlt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 00:52:26.382971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.382736 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 00:52:26.442747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.441954 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:26.474424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.474017 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 00:52:26.474424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.474118 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 00:52:26.970020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:26.969251 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:52:27.038950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.038850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:27.181987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.117074 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:52:27.181987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.117215 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.181987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.137029 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fa555-openshift-default-dockercfg-6sx7q\"" Oct 30 00:52:27.279965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.279665 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjnc\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.279965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.279838 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.279965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.279899 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.279965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.279944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.279965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.279969 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.284658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.280043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.284658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.280117 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.284658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.284079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.284658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.284184 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411114 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411257 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411339 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411368 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411412 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411496 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.411689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjnc\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.412679 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.413388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.413665 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.413953 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.414503 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.433926 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.452671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.518854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.453570 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjnc\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.529179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.527119 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:27.591099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.590981 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\") pod \"e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.836889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.836088 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:27.969982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:27.969582 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:28.122329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.122299 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:28.124050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.124029 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:28.127132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.127109 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 00:52:28.380276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.380172 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:28.382111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.382092 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:28.385506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.385487 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 00:52:28.457366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.457019 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:28.457366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.457083 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:28.518313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.518079 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kxdph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:52:28.518313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.518134 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:52:28.683612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.683576 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:52:28.686629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.686587 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:28.686726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.686672 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:28.686726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.686712 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:28.855594 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:52:28.855515 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:52:28.876877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.876826 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:28.878336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.878303 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:28.879216 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:28.879188 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 00:52:29.046117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.045898 5938 scope.go:122] "RemoveContainer" containerID="0e0b57c12533673f77ea3ce120d2554d56957e91eb018c29295bb68508b29b97" Oct 30 00:52:29.062604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.062407 5938 scope.go:122] "RemoveContainer" containerID="191f0166b723c2f50e9ea3a6263661640032c64056d240889575048cb0f4f3b2" Oct 30 00:52:29.703937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.703899 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerStarted","Data":"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111"} Oct 30 00:52:29.704993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.703962 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerStarted","Data":"4126973fc49f41a0ac8f3db864970332a9ca4e051c936c3d85d748389989f67f"} Oct 30 00:52:29.754451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.754393 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podStartSLOduration=3.75437558 podStartE2EDuration="3.75437558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:29.747263464 +0000 UTC m=+1201.987498627" watchObservedRunningTime="2026-10-30 00:52:29.75437558 +0000 UTC m=+1201.994610697" Oct 30 00:52:29.836638 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.836600 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:29.844235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.844206 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:29.876924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:29.876895 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:30.336956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:30.336919 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:52:30.363195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:30.363171 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:52:30.713033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:30.713000 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:30.725649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:30.713916 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:52:31.562165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.562128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:31.722787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.722753 5938 generic.go:356] "Generic (PLEG): container finished" podID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerID="f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a" exitCode=0 Oct 30 00:52:31.723344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.722829 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerDied","Data":"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a"} Oct 30 00:52:31.723408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.723393 5938 scope.go:122] "RemoveContainer" containerID="f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a" Oct 30 00:52:31.724751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.724729 5938 generic.go:356] "Generic (PLEG): container finished" podID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerID="1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d" exitCode=0 Oct 30 00:52:31.724897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.724805 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerDied","Data":"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d"} Oct 30 00:52:31.725231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.725199 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:31.725326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.725266 5938 scope.go:122] "RemoveContainer" containerID="1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d" Oct 30 00:52:31.725326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.725271 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:31.725326 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.725307 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:31.726636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.726608 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:31.726729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.726686 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:31.726794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:31.726749 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:32.176238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:32.176155 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:32.346885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:32.346811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:32.742846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:32.742815 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerStarted","Data":"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726"} Oct 30 00:52:32.746481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:32.746444 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerStarted","Data":"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17"} Oct 30 00:52:33.504894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.503414 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:33.583321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.573533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:52:33.583321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.573717 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:52:33.583321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.573767 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:52:33.675465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.675434 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:52:33.675633 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.675605 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:52:33.675685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:33.675651 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:52:34.039903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.025689 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.205895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.184101 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.558631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.558594 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:52:34.558850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.558674 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:52:34.676647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.676611 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:52:34.676838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:34.676657 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:52:35.553937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:35.553904 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:52:35.554377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:35.553957 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:52:35.562546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:35.562516 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:35.698888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:35.697979 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:52:35.698888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:35.698162 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:52:36.501901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:36.501151 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:36.555196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:36.555142 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:52:36.555673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:36.555250 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:52:36.676151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:36.676111 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:52:36.676312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:36.676167 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:52:37.110914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.107931 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:37.553484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.553403 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.553484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.553463 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 00:52:37.651514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.651483 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:52:37.675936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.675909 5938 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 00:52:37.676082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:37.675952 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 00:52:38.557690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.557663 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:52:38.558843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.558821 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 00:52:38.621035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.621007 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:38.683996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.683236 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:52:38.685000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.684963 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:52:38.693230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:38.686257 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:52:39.030343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:39.030311 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz"] Oct 30 00:52:39.034788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:39.034760 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46fea2a2-tqjnz"] Oct 30 00:52:39.877519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:39.877489 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f0de89-a598-4902-9b83-89a274e87d44" path="/var/lib/kubelet/pods/f0f0de89-a598-4902-9b83-89a274e87d44/volumes" Oct 30 00:52:40.373659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:40.373625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:41.869220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:41.869186 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:43.439855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:43.439819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:45.579384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:45.579348 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:45.727886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:45.724725 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.045651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:47.045616 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l"] Oct 30 00:52:47.056481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:47.056454 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4afd8c5-lll9l"] Oct 30 00:52:47.607907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:47.607850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:47.878721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:47.878642 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96" path="/var/lib/kubelet/pods/7453aa9e-5a43-4b10-a5ae-22a0b7aa8b96/volumes" Oct 30 00:52:49.239427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:49.239353 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:50.817724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:50.817688 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:52.314149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:52.314119 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:52.484134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:52.484101 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:53.030083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.030044 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd"] Oct 30 00:52:53.042156 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.042123 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced525e4-dhkpd"] Oct 30 00:52:53.354905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.354857 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-422b08"] Oct 30 00:52:53.362231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.362210 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:52:53.377001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.376931 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-422b08"] Oct 30 00:52:53.396651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.396626 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\"" Oct 30 00:52:53.455260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.455228 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk7l\" (UniqueName: \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\") pod \"test-curl-8804-422b08\" (UID: \"d8b8eef0-acff-45a4-9aa5-0390136aac91\") " pod="opendatahub/test-curl-8804-422b08" Oct 30 00:52:53.556724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.556685 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk7l\" (UniqueName: \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\") pod \"test-curl-8804-422b08\" (UID: \"d8b8eef0-acff-45a4-9aa5-0390136aac91\") " pod="opendatahub/test-curl-8804-422b08" Oct 30 00:52:53.580544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.580512 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk7l\" (UniqueName: \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\") pod \"test-curl-8804-422b08\" (UID: \"d8b8eef0-acff-45a4-9aa5-0390136aac91\") " pod="opendatahub/test-curl-8804-422b08" Oct 30 00:52:53.686478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.686400 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:52:53.884032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.883999 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bb96e3-cdc9-4ea3-a633-2cb44101a84a" path="/var/lib/kubelet/pods/18bb96e3-cdc9-4ea3-a633-2cb44101a84a/volumes" Oct 30 00:52:53.936708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:53.936633 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:54.021408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:54.021322 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-422b08"] Oct 30 00:52:54.040652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:54.040622 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:52:54.069836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:54.069812 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:55.103211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:55.103141 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-422b08" event={"ID":"d8b8eef0-acff-45a4-9aa5-0390136aac91","Type":"ContainerStarted","Data":"fa99ac937fcad80fa5828140d81922ce2e1edf6dd322baf1c618604ff8f739dc"} Oct 30 00:52:55.705309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:55.705267 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:56.112323 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.112291 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-422b08" event={"ID":"d8b8eef0-acff-45a4-9aa5-0390136aac91","Type":"ContainerStarted","Data":"7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992"} Oct 30 00:52:56.114161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.114138 5938 generic.go:356] "Generic (PLEG): container finished" podID="6982266f-1fbe-485d-94bf-a8267348734b" containerID="c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3" exitCode=0 Oct 30 00:52:56.114283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.114217 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerDied","Data":"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3"} Oct 30 00:52:56.114635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.114618 5938 scope.go:122] "RemoveContainer" containerID="c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3" Oct 30 00:52:56.116444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.116410 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:56.116549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.116498 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:56.116622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.116553 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:52:56.152012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:56.151971 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-422b08" podStartSLOduration=2.30718048 podStartE2EDuration="3.151959027s" totalImagesPullingTime="844.778547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:56.148192207 +0000 UTC m=+1228.388427314" watchObservedRunningTime="2026-10-30 00:52:56.151959027 +0000 UTC m=+1228.392194134" Oct 30 00:52:57.121301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.121270 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-422b08"] Oct 30 00:52:57.128632 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.128600 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-422b08"] Oct 30 00:52:57.130371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.130342 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerStarted","Data":"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1"} Oct 30 00:52:57.260491 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.260456 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-96d0ff"] Oct 30 00:52:57.261185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.261169 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" containerName="test-curl-8804-422b08" Oct 30 00:52:57.261248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.261188 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" containerName="test-curl-8804-422b08" Oct 30 00:52:57.261348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.261339 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" containerName="test-curl-8804-422b08" Oct 30 00:52:57.264158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.264140 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:52:57.280905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.280835 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-96d0ff"] Oct 30 00:52:57.340169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.340141 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:57.370948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.364001 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:52:57.370948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.366575 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvb6\" (UniqueName: \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\") pod \"test-curl-8804-96d0ff\" (UID: \"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e\") " pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:52:57.370948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.368024 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:52:57.370948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.368078 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:52:57.468479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.468388 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvb6\" (UniqueName: \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\") pod \"test-curl-8804-96d0ff\" (UID: \"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e\") " pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:52:57.510346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.510317 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvb6\" (UniqueName: \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\") pod \"test-curl-8804-96d0ff\" (UID: \"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e\") " pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:52:57.577077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.577050 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:52:57.915637 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:52:57.915603 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1884fe3_e8fc_45f3_9c47_f82b84cb0f8e.slice/crio-873a46462857edb425f7000f33afdf41531036e4fc1689ab1c1cb2fde759bf81 WatchSource:0}: Error finding container 873a46462857edb425f7000f33afdf41531036e4fc1689ab1c1cb2fde759bf81: Status 404 returned error can't find the container with id 873a46462857edb425f7000f33afdf41531036e4fc1689ab1c1cb2fde759bf81 Oct 30 00:52:57.915967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:57.915948 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-96d0ff"] Oct 30 00:52:58.140803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.140769 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-96d0ff" event={"ID":"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e","Type":"ContainerStarted","Data":"a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725"} Oct 30 00:52:58.141242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.140820 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-96d0ff" event={"ID":"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e","Type":"ContainerStarted","Data":"873a46462857edb425f7000f33afdf41531036e4fc1689ab1c1cb2fde759bf81"} Oct 30 00:52:58.141242 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.140935 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-422b08" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" containerName="test-curl-8804-422b08" containerID="cri-o://7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992" gracePeriod=2 Oct 30 00:52:58.182533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.182461 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-96d0ff" podStartSLOduration=1.182447682 podStartE2EDuration="1.182447682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:58.177887571 +0000 UTC m=+1230.418122701" watchObservedRunningTime="2026-10-30 00:52:58.182447682 +0000 UTC m=+1230.422682792" Oct 30 00:52:58.353848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.353816 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:52:58.354060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.353919 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:52:58.764820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.764789 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw"] Oct 30 00:52:58.768384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.768362 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.782757 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.782734 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:52:58.782918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.782821 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:52:58.782918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.782904 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:52:58.783032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.782920 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:52:58.813955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.813922 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw"] Oct 30 00:52:58.818809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.818779 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.818988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.818922 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxl9\" (UniqueName: \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.818988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.818964 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.847722 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:52:58.847690 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:52:58.920472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.920441 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.920678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.920542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxl9\" (UniqueName: \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.920678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.920577 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.921143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.921123 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.922974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.922950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:58.942983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:58.942960 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxl9\" (UniqueName: \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\") pod \"maas-api-revoke-sub-86f0071b-cxhkw\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:59.048523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.048443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:59.084893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.082324 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:52:59.228133 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.227521 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-96d0ff"] Oct 30 00:52:59.236306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.236278 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-96d0ff"] Oct 30 00:52:59.354070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.354005 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:52:59.354262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.354072 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:52:59.367032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.367001 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-f281be"] Oct 30 00:52:59.368215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.368192 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" containerName="test-curl-8804-96d0ff" Oct 30 00:52:59.368334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.368216 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" containerName="test-curl-8804-96d0ff" Oct 30 00:52:59.368433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.368414 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" containerName="test-curl-8804-96d0ff" Oct 30 00:52:59.372277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.372260 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-f281be" Oct 30 00:52:59.382210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.382185 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-f281be"] Oct 30 00:52:59.434069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.434034 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zgw\" (UniqueName: \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\") pod \"test-curl-8804-f281be\" (UID: \"1ad51256-eda1-4aab-938c-1b07c4ac5ff5\") " pod="opendatahub/test-curl-8804-f281be" Oct 30 00:52:59.513704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.513663 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw"] Oct 30 00:52:59.514773 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:52:59.514747 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51dfce_b0d5_49de_abb2_53c6fe009d6e.slice/crio-b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd WatchSource:0}: Error finding container b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd: Status 404 returned error can't find the container with id b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd Oct 30 00:52:59.535482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.535454 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zgw\" (UniqueName: \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\") pod \"test-curl-8804-f281be\" (UID: \"1ad51256-eda1-4aab-938c-1b07c4ac5ff5\") " pod="opendatahub/test-curl-8804-f281be" Oct 30 00:52:59.557548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.557528 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zgw\" (UniqueName: \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\") pod \"test-curl-8804-f281be\" (UID: \"1ad51256-eda1-4aab-938c-1b07c4ac5ff5\") " pod="opendatahub/test-curl-8804-f281be" Oct 30 00:52:59.716646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.716579 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-f281be" Oct 30 00:52:59.906977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:52:59.896945 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:53:00.028566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.028529 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-f281be"] Oct 30 00:53:00.029259 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:00.029227 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ad51256_eda1_4aab_938c_1b07c4ac5ff5.slice/crio-2b0ebe07a3023dc8a2499f907ab1e46499c2f776a8232819d2454f3fa6c020d5 WatchSource:0}: Error finding container 2b0ebe07a3023dc8a2499f907ab1e46499c2f776a8232819d2454f3fa6c020d5: Status 404 returned error can't find the container with id 2b0ebe07a3023dc8a2499f907ab1e46499c2f776a8232819d2454f3fa6c020d5 Oct 30 00:53:00.177365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.177221 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" event={"ID":"bf51dfce-b0d5-49de-abb2-53c6fe009d6e","Type":"ContainerStarted","Data":"216f1ff5c6e9ffe3f1994c6f713efb240df02dcc2729418d9daf1f92a4a29530"} Oct 30 00:53:00.177365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.177326 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" event={"ID":"bf51dfce-b0d5-49de-abb2-53c6fe009d6e","Type":"ContainerStarted","Data":"b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd"} Oct 30 00:53:00.183443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.183411 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-f281be" event={"ID":"1ad51256-eda1-4aab-938c-1b07c4ac5ff5","Type":"ContainerStarted","Data":"2eb216b83d6ce01018e69a0b29b03083e9abcae6c052a2a49655bc1cd3c0cf9c"} Oct 30 00:53:00.183552 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.183457 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-f281be" event={"ID":"1ad51256-eda1-4aab-938c-1b07c4ac5ff5","Type":"ContainerStarted","Data":"2b0ebe07a3023dc8a2499f907ab1e46499c2f776a8232819d2454f3fa6c020d5"} Oct 30 00:53:00.183618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.183574 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-96d0ff" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" containerName="test-curl-8804-96d0ff" containerID="cri-o://a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725" gracePeriod=2 Oct 30 00:53:00.223320 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:00.223293 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8b8eef0_acff_45a4_9aa5_0390136aac91.slice/crio-7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8b8eef0_acff_45a4_9aa5_0390136aac91.slice/crio-conmon-7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992.scope\": RecentStats: unable to find data in memory cache]" Oct 30 00:53:00.242477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.242430 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" podStartSLOduration=2.242415786 podStartE2EDuration="2.242415786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:00.235255865 +0000 UTC m=+1232.475490992" watchObservedRunningTime="2026-10-30 00:53:00.242415786 +0000 UTC m=+1232.482650911" Oct 30 00:53:00.278345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.278254 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-f281be" podStartSLOduration=1.278240744 podStartE2EDuration="1.278240744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:00.272725597 +0000 UTC m=+1232.512960716" watchObservedRunningTime="2026-10-30 00:53:00.278240744 +0000 UTC m=+1232.518475856" Oct 30 00:53:00.354723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.353791 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:53:00.354723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.353852 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:53:00.432791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.432768 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:53:00.553131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.553070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\" (UniqueName: \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\") pod \"d8b8eef0-acff-45a4-9aa5-0390136aac91\" (UID: \"d8b8eef0-acff-45a4-9aa5-0390136aac91\") " Oct 30 00:53:00.555143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.555119 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l" pod "d8b8eef0-acff-45a4-9aa5-0390136aac91" (UID: "d8b8eef0-acff-45a4-9aa5-0390136aac91"). InnerVolumeSpecName "kube-api-access-9vk7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:00.654913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.654888 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vk7l\" (UniqueName: \"kubernetes.io/projected/d8b8eef0-acff-45a4-9aa5-0390136aac91-kube-api-access-9vk7l\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:00.701887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:00.695199 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:01.221014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.220474 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-f281be"] Oct 30 00:53:01.224890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.224375 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-f281be"] Oct 30 00:53:01.224890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.224513 5938 generic.go:356] "Generic (PLEG): container finished" podID="d8b8eef0-acff-45a4-9aa5-0390136aac91" containerID="7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992" exitCode=137 Oct 30 00:53:01.224890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.224558 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:53:01.224890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.224612 5938 scope.go:122] "RemoveContainer" containerID="7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992" Oct 30 00:53:01.230409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.229331 5938 generic.go:356] "Generic (PLEG): container finished" podID="bf51dfce-b0d5-49de-abb2-53c6fe009d6e" containerID="216f1ff5c6e9ffe3f1994c6f713efb240df02dcc2729418d9daf1f92a4a29530" exitCode=0 Oct 30 00:53:01.230409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.230304 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" event={"ID":"bf51dfce-b0d5-49de-abb2-53c6fe009d6e","Type":"ContainerDied","Data":"216f1ff5c6e9ffe3f1994c6f713efb240df02dcc2729418d9daf1f92a4a29530"} Oct 30 00:53:01.243171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.243151 5938 scope.go:122] "RemoveContainer" containerID="7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992" Oct 30 00:53:01.243503 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:01.243477 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992\": container with ID starting with 7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992 not found: ID does not exist" containerID="7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992" Oct 30 00:53:01.243562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.243517 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992"} err="failed to get container status \"7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992\": rpc error: code = NotFound desc = could not find container \"7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992\": container with ID starting with 7e262c95ec6471272f4eeadb42733293d064b0b169907e9f8e0ace944e29a992 not found: ID does not exist" Oct 30 00:53:01.311943 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:01.311104 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-422b08\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:53:01.353969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.353942 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 00:53:01.354127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.353989 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 00:53:01.373895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.373838 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-84ac65"] Oct 30 00:53:01.374547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.374531 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" containerName="test-curl-8804-f281be" Oct 30 00:53:01.374588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.374550 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" containerName="test-curl-8804-f281be" Oct 30 00:53:01.374718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.374708 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" containerName="test-curl-8804-f281be" Oct 30 00:53:01.377925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.377908 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:01.419721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.419692 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-84ac65"] Oct 30 00:53:01.448054 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:01.448025 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-422b08\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" pod="opendatahub/test-curl-8804-422b08" Oct 30 00:53:01.466758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.466732 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhrs\" (UniqueName: \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\") pod \"test-curl-8804-84ac65\" (UID: \"7db8f52d-eea4-43e5-8315-b4df0651c6e0\") " pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:01.568597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.568532 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhrs\" (UniqueName: \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\") pod \"test-curl-8804-84ac65\" (UID: \"7db8f52d-eea4-43e5-8315-b4df0651c6e0\") " pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:01.609973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.609944 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhrs\" (UniqueName: \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\") pod \"test-curl-8804-84ac65\" (UID: \"7db8f52d-eea4-43e5-8315-b4df0651c6e0\") " pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:01.720019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.719988 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:01.879227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:01.879199 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b8eef0-acff-45a4-9aa5-0390136aac91" path="/var/lib/kubelet/pods/d8b8eef0-acff-45a4-9aa5-0390136aac91/volumes" Oct 30 00:53:02.025145 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:02.025104 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db8f52d_eea4_43e5_8315_b4df0651c6e0.slice/crio-ffffed84588085ae147fee9927f7620d6dfe24251cb0ffc2284194a97a01fd3b WatchSource:0}: Error finding container ffffed84588085ae147fee9927f7620d6dfe24251cb0ffc2284194a97a01fd3b: Status 404 returned error can't find the container with id ffffed84588085ae147fee9927f7620d6dfe24251cb0ffc2284194a97a01fd3b Oct 30 00:53:02.025145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.025110 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-84ac65"] Oct 30 00:53:02.186031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.185962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:02.243173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.243141 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-84ac65" event={"ID":"7db8f52d-eea4-43e5-8315-b4df0651c6e0","Type":"ContainerStarted","Data":"968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf"} Oct 30 00:53:02.243586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.243194 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-84ac65" event={"ID":"7db8f52d-eea4-43e5-8315-b4df0651c6e0","Type":"ContainerStarted","Data":"ffffed84588085ae147fee9927f7620d6dfe24251cb0ffc2284194a97a01fd3b"} Oct 30 00:53:02.244755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.244620 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-f281be" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" containerName="test-curl-8804-f281be" containerID="cri-o://2eb216b83d6ce01018e69a0b29b03083e9abcae6c052a2a49655bc1cd3c0cf9c" gracePeriod=2 Oct 30 00:53:02.436681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.435258 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:53:02.442334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.442217 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 00:53:02.489278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.489222 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-84ac65" podStartSLOduration=1.489202285 podStartE2EDuration="1.489202285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:02.292124684 +0000 UTC m=+1234.532359823" watchObservedRunningTime="2026-10-30 00:53:02.489202285 +0000 UTC m=+1234.729437400" Oct 30 00:53:02.545933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.545468 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:53:02.574177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.574148 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:53:02.614688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.614657 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\" (UniqueName: \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\") pod \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " Oct 30 00:53:02.615035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.615010 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\") pod \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " Oct 30 00:53:02.615464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.615437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca" pod "bf51dfce-b0d5-49de-abb2-53c6fe009d6e" (UID: "bf51dfce-b0d5-49de-abb2-53c6fe009d6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:02.615643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.615622 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\") pod \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\" (UID: \"bf51dfce-b0d5-49de-abb2-53c6fe009d6e\") " Oct 30 00:53:02.616608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.616580 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:02.619170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.619142 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca" pod "bf51dfce-b0d5-49de-abb2-53c6fe009d6e" (UID: "bf51dfce-b0d5-49de-abb2-53c6fe009d6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:02.620421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.620391 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9" pod "bf51dfce-b0d5-49de-abb2-53c6fe009d6e" (UID: "bf51dfce-b0d5-49de-abb2-53c6fe009d6e"). InnerVolumeSpecName "kube-api-access-5pxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:02.717426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.717341 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\" (UniqueName: \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\") pod \"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e\" (UID: \"f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e\") " Oct 30 00:53:02.717822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.717807 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:02.717822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.717824 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pxl9\" (UniqueName: \"kubernetes.io/projected/bf51dfce-b0d5-49de-abb2-53c6fe009d6e-kube-api-access-5pxl9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:02.719478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.719455 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6" pod "f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" (UID: "f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e"). InnerVolumeSpecName "kube-api-access-zdvb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:02.819615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:02.819583 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdvb6\" (UniqueName: \"kubernetes.io/projected/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e-kube-api-access-zdvb6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:03.187143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.187117 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-84ac65"] Oct 30 00:53:03.197913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.194339 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-84ac65"] Oct 30 00:53:03.256515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.256478 5938 generic.go:356] "Generic (PLEG): container finished" podID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" containerID="a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725" exitCode=137 Oct 30 00:53:03.257020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.256533 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:53:03.257282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.256539 5938 scope.go:122] "RemoveContainer" containerID="a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725" Oct 30 00:53:03.265824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.265787 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" Oct 30 00:53:03.265965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.265852 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw" event={"ID":"bf51dfce-b0d5-49de-abb2-53c6fe009d6e","Type":"ContainerDied","Data":"b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd"} Oct 30 00:53:03.265965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.265949 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17af8d965af1176f7dc53a7eef0c5135ef55f4f6ecec6512c885d4ff43c64cd" Oct 30 00:53:03.279008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.278991 5938 scope.go:122] "RemoveContainer" containerID="a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725" Oct 30 00:53:03.279547 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:03.279517 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725\": container with ID starting with a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725 not found: ID does not exist" containerID="a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725" Oct 30 00:53:03.279673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.279556 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725"} err="failed to get container status \"a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725\": rpc error: code = NotFound desc = could not find container \"a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725\": container with ID starting with a0110ed296fe779608e9b675b12a435ae028002bd0c4fd7f2b3d611e8dd4e725 not found: ID does not exist" Oct 30 00:53:03.280607 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:03.280578 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-96d0ff\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:53:03.294461 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:03.294410 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-96d0ff\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" pod="opendatahub/test-curl-8804-96d0ff" Oct 30 00:53:03.786356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.786312 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:03.877685 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.877653 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e" path="/var/lib/kubelet/pods/f1884fe3-e8fc-45f3-9c47-f82b84cb0f8e/volumes" Oct 30 00:53:03.949478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:03.949453 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:04.031624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.031597 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp"] Oct 30 00:53:04.043618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.043520 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a719eeaa-54xsp"] Oct 30 00:53:04.174719 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.174686 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:53:04.175902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.175840 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bf51dfce-b0d5-49de-abb2-53c6fe009d6e" containerName="revoke-keys" Oct 30 00:53:04.176021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.175904 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51dfce-b0d5-49de-abb2-53c6fe009d6e" containerName="revoke-keys" Oct 30 00:53:04.176021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.175955 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" containerName="test-curl-8804-84ac65" Oct 30 00:53:04.176021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.175963 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" containerName="test-curl-8804-84ac65" Oct 30 00:53:04.176141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.176128 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" containerName="test-curl-8804-84ac65" Oct 30 00:53:04.176191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.176157 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51dfce-b0d5-49de-abb2-53c6fe009d6e" containerName="revoke-keys" Oct 30 00:53:04.180523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.180501 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.201508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.201481 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:53:04.238188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238154 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238388 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238418 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8sm\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238483 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238525 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238568 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.238913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238828 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.239078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.238910 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.257559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.257532 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:04.274412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.274349 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d788ba-openshift-default-dockercfg-sd65d\"" Oct 30 00:53:04.374941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.345153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.375235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.375209 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.375725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.375702 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8sm\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.375969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.375948 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.376134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.376117 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.376264 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.376248 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.396885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.376696 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.397212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.397182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.397531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.397512 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.398181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.398156 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.398333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.372353 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.418885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.399383 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.419090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.408536 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.419198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.409191 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.430895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.426971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.454889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.442625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:04.468736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.466511 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8sm\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.468736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.466754 5938 generic.go:356] "Generic (PLEG): container finished" podID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" containerID="2eb216b83d6ce01018e69a0b29b03083e9abcae6c052a2a49655bc1cd3c0cf9c" exitCode=137 Oct 30 00:53:04.484435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.484376 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.485356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.484713 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-84ac65" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" containerName="test-curl-8804-84ac65" containerID="cri-o://968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf" gracePeriod=2 Oct 30 00:53:04.495991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.495910 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\") pod \"e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.505440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.505187 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:04.803652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.803620 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-f281be" Oct 30 00:53:04.922219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.922157 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\" (UniqueName: \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\") pod \"1ad51256-eda1-4aab-938c-1b07c4ac5ff5\" (UID: \"1ad51256-eda1-4aab-938c-1b07c4ac5ff5\") " Oct 30 00:53:04.924211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.924190 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw" pod "1ad51256-eda1-4aab-938c-1b07c4ac5ff5" (UID: "1ad51256-eda1-4aab-938c-1b07c4ac5ff5"). InnerVolumeSpecName "kube-api-access-t5zgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:04.962914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.962885 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:53:04.963982 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:04.963959 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode369062e_6423_4537_91fa_55edba20c3af.slice/crio-89a78edcd6e98e73c516338dd84a2d9aeea0569475d98f5c377133978da90435 WatchSource:0}: Error finding container 89a78edcd6e98e73c516338dd84a2d9aeea0569475d98f5c377133978da90435: Status 404 returned error can't find the container with id 89a78edcd6e98e73c516338dd84a2d9aeea0569475d98f5c377133978da90435 Oct 30 00:53:04.966455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.966425 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:04.966529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.966500 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:04.966569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:04.966532 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:05.025336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.025309 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zgw\" (UniqueName: \"kubernetes.io/projected/1ad51256-eda1-4aab-938c-1b07c4ac5ff5-kube-api-access-t5zgw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:05.495477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.495439 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerStarted","Data":"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f"} Oct 30 00:53:05.495948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.495536 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerStarted","Data":"89a78edcd6e98e73c516338dd84a2d9aeea0569475d98f5c377133978da90435"} Oct 30 00:53:05.497116 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.497092 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-f281be" Oct 30 00:53:05.497243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.497193 5938 scope.go:122] "RemoveContainer" containerID="2eb216b83d6ce01018e69a0b29b03083e9abcae6c052a2a49655bc1cd3c0cf9c" Oct 30 00:53:05.506818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.506800 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:05.541729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.541660 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podStartSLOduration=1.541648949 podStartE2EDuration="1.541648949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:05.538744224 +0000 UTC m=+1237.778979338" watchObservedRunningTime="2026-10-30 00:53:05.541648949 +0000 UTC m=+1237.781884122" Oct 30 00:53:05.551735 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:05.551711 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-f281be\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" pod="opendatahub/test-curl-8804-f281be" Oct 30 00:53:05.738065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.738033 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:05.878640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.878597 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad51256-eda1-4aab-938c-1b07c4ac5ff5" path="/var/lib/kubelet/pods/1ad51256-eda1-4aab-938c-1b07c4ac5ff5/volumes" Oct 30 00:53:05.879348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:05.879323 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75c5579-9aba-4b05-89e1-c87f3195e417" path="/var/lib/kubelet/pods/c75c5579-9aba-4b05-89e1-c87f3195e417/volumes" Oct 30 00:53:06.512426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:06.512400 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:06.733351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:06.733327 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:06.749094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:06.749067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\" (UniqueName: \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\") pod \"7db8f52d-eea4-43e5-8315-b4df0651c6e0\" (UID: \"7db8f52d-eea4-43e5-8315-b4df0651c6e0\") " Oct 30 00:53:06.751051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:06.751025 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs" pod "7db8f52d-eea4-43e5-8315-b4df0651c6e0" (UID: "7db8f52d-eea4-43e5-8315-b4df0651c6e0"). InnerVolumeSpecName "kube-api-access-cjhrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:06.850514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:06.850454 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhrs\" (UniqueName: \"kubernetes.io/projected/7db8f52d-eea4-43e5-8315-b4df0651c6e0-kube-api-access-cjhrs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:07.447677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.447641 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:07.523643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.523573 5938 generic.go:356] "Generic (PLEG): container finished" podID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" containerID="968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf" exitCode=137 Oct 30 00:53:07.523643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.523633 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:07.524225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.523689 5938 scope.go:122] "RemoveContainer" containerID="968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf" Oct 30 00:53:07.524564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.524535 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:07.527492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.527138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:53:07.544844 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.544825 5938 scope.go:122] "RemoveContainer" containerID="968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf" Oct 30 00:53:07.545226 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:07.545202 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf\": container with ID starting with 968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf not found: ID does not exist" containerID="968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf" Oct 30 00:53:07.545297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.545234 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf"} err="failed to get container status \"968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf\": rpc error: code = NotFound desc = could not find container \"968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf\": container with ID starting with 968162cc0067d9fd8c3fd07a90d0c67c3d1627a9f08fe31535b475ae13c32adf not found: ID does not exist" Oct 30 00:53:07.594701 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:07.594664 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-84ac65\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:07.878346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:07.878316 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" path="/var/lib/kubelet/pods/7db8f52d-eea4-43e5-8315-b4df0651c6e0/volumes" Oct 30 00:53:07.887066 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:07.887031 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-84ac65\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="7db8f52d-eea4-43e5-8315-b4df0651c6e0" pod="opendatahub/test-curl-8804-84ac65" Oct 30 00:53:08.051221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:08.051193 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd"] Oct 30 00:53:08.056840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:08.056811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:08.058309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:08.058286 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63f65bdf-xr8wd"] Oct 30 00:53:08.201101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:08.201020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.203886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.202744 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.338579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.338546 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.476817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.476741 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.622986 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.622950 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.866891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.866842 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:09.878974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.878945 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7078218e-0204-439a-8d2e-19847cdcc867" path="/var/lib/kubelet/pods/7078218e-0204-439a-8d2e-19847cdcc867/volumes" Oct 30 00:53:09.954605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.954577 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:53:09.958548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.958532 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:09.976225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.976203 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:53:09.982644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.982618 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d788ba\"" Oct 30 00:53:09.982644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:09.982636 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d788ba-dockercfg-scwsm\"" Oct 30 00:53:10.016105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.016073 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:10.045035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.045012 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ncm\" (UniqueName: \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.045228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.045209 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.152998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.149527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ncm\" (UniqueName: \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.152998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.149698 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.191891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.176693 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.365900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.362099 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:53:10.381895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.367276 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:53:10.381895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.367394 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.381895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.378401 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ncm\" (UniqueName: \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\") pod \"payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.450032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.449593 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.450032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.449785 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn4z\" (UniqueName: \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.550957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.550910 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.551150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.551008 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn4z\" (UniqueName: \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.551599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.551577 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.571855 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.571831 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:10.574290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.573959 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn4z\" (UniqueName: \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\") pod \"payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.748232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.748159 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:10.888331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:10.888304 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:53:11.058051 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.058019 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:53:11.058412 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:11.058389 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43168dae_5f77_4d46_ba9a_098b000abea5.slice/crio-ba815f5be8b967d079bd928b0c89ae4af3b0891cea231d27205a59f01075aa0a WatchSource:0}: Error finding container ba815f5be8b967d079bd928b0c89ae4af3b0891cea231d27205a59f01075aa0a: Status 404 returned error can't find the container with id ba815f5be8b967d079bd928b0c89ae4af3b0891cea231d27205a59f01075aa0a Oct 30 00:53:11.061343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.061315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:11.229897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.228121 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:11.563203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.563166 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" event={"ID":"43168dae-5f77-4d46-ba9a-098b000abea5","Type":"ContainerStarted","Data":"4fc77ccf8846018d0701a43f681c19ef6de9474d35a85ba249f86e52a575e76a"} Oct 30 00:53:11.563618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.563228 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" event={"ID":"43168dae-5f77-4d46-ba9a-098b000abea5","Type":"ContainerStarted","Data":"ba815f5be8b967d079bd928b0c89ae4af3b0891cea231d27205a59f01075aa0a"} Oct 30 00:53:11.563618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.563257 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:11.565274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.565244 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" event={"ID":"055dfa30-388c-463a-88c2-e3022034c5cb","Type":"ContainerStarted","Data":"143be4f7fb41a74084dd3f8f850f7a41f73f323f10c4662107536ee50513e33c"} Oct 30 00:53:11.565405 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.565294 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:11.565405 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.565313 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" event={"ID":"055dfa30-388c-463a-88c2-e3022034c5cb","Type":"ContainerStarted","Data":"c8868287bd4b61f4275c97b217057e4f21d52e78cac16cb5858fdf7280bd51ce"} Oct 30 00:53:11.603762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.603720 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" podStartSLOduration=1.603707188 podStartE2EDuration="1.603707188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.601582369 +0000 UTC m=+1243.841817509" watchObservedRunningTime="2026-10-30 00:53:11.603707188 +0000 UTC m=+1243.843942312" Oct 30 00:53:11.644083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:11.644049 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" podStartSLOduration=2.644038298 podStartE2EDuration="2.644038298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:11.640056529 +0000 UTC m=+1243.880291842" watchObservedRunningTime="2026-10-30 00:53:11.644038298 +0000 UTC m=+1243.884273401" Oct 30 00:53:12.029573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.029488 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql"] Oct 30 00:53:12.033749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.033728 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8afe1aed-sspql"] Oct 30 00:53:12.187945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.187914 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.353753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.353722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.507406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.507379 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.851981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:12.851956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:13.879688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:13.879654 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9122261f-ea39-4ed3-834c-a4e0c7a2aa0d" path="/var/lib/kubelet/pods/9122261f-ea39-4ed3-834c-a4e0c7a2aa0d/volumes" Oct 30 00:53:15.204891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:15.204170 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:15.822985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:15.822939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:16.781435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:16.781395 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:17.029313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:17.029278 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7"] Oct 30 00:53:17.042090 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:17.042026 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da53c58-z44k7"] Oct 30 00:53:17.878964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:17.878930 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa2fc28-76d9-4e4e-bbed-7fd216cc6270" path="/var/lib/kubelet/pods/4fa2fc28-76d9-4e4e-bbed-7fd216cc6270/volumes" Oct 30 00:53:18.480096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:18.480064 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:20.227410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:20.227373 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:21.911054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:21.911022 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:22.577418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.577379 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:53:22.577608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.577453 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:53:22.914532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.914498 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8807-6e2730"] Oct 30 00:53:22.918401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.918378 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:22.932479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.932444 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\"" Oct 30 00:53:22.935092 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:22.935070 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8807-6e2730"] Oct 30 00:53:23.033907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.033824 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86cb\" (UniqueName: \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\") pod \"test-curl-8807-6e2730\" (UID: \"b64f1e92-f473-4f47-aa44-2ea48011a677\") " pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:23.135291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.135241 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86cb\" (UniqueName: \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\") pod \"test-curl-8807-6e2730\" (UID: \"b64f1e92-f473-4f47-aa44-2ea48011a677\") " pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:23.143849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.143819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:23.172470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.172398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86cb\" (UniqueName: \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\") pod \"test-curl-8807-6e2730\" (UID: \"b64f1e92-f473-4f47-aa44-2ea48011a677\") " pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:23.232949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.232918 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:23.515540 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.515462 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:23.581397 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:23.581360 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb64f1e92_f473_4f47_aa44_2ea48011a677.slice/crio-9c573c21c30361d20d931d54d13052434d42734ddd0655939deb8292026ac9df WatchSource:0}: Error finding container 9c573c21c30361d20d931d54d13052434d42734ddd0655939deb8292026ac9df: Status 404 returned error can't find the container with id 9c573c21c30361d20d931d54d13052434d42734ddd0655939deb8292026ac9df Oct 30 00:53:23.582271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.582244 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8807-6e2730"] Oct 30 00:53:23.753266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.753231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8807-6e2730" event={"ID":"b64f1e92-f473-4f47-aa44-2ea48011a677","Type":"ContainerStarted","Data":"31b8625bbb7f1e7a827abb4b3737694422b853e561a9f43c7b699b461e85d770"} Oct 30 00:53:23.753376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.753278 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8807-6e2730" event={"ID":"b64f1e92-f473-4f47-aa44-2ea48011a677","Type":"ContainerStarted","Data":"9c573c21c30361d20d931d54d13052434d42734ddd0655939deb8292026ac9df"} Oct 30 00:53:23.811162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.811082 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8807-6e2730" podStartSLOduration=1.811067653 podStartE2EDuration="1.811067653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:23.805269891 +0000 UTC m=+1256.045505007" watchObservedRunningTime="2026-10-30 00:53:23.811067653 +0000 UTC m=+1256.051302765" Oct 30 00:53:23.851027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:23.850999 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:24.409827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:24.409792 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:24.565885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:24.565355 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:25.046577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.046546 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8807-6e2730"] Oct 30 00:53:25.052631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.052601 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8807-6e2730"] Oct 30 00:53:25.378248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.378214 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:25.665204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.665131 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx"] Oct 30 00:53:25.666071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.666053 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" containerName="test-curl-8807-6e2730" Oct 30 00:53:25.666138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.666073 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" containerName="test-curl-8807-6e2730" Oct 30 00:53:25.666198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.666187 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" containerName="test-curl-8807-6e2730" Oct 30 00:53:25.670009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.669994 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.709922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.707125 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:25.709922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.707225 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:25.709922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.707745 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:25.709922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.707902 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:53:25.710275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.709975 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx"] Oct 30 00:53:25.833207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.833169 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vq8t\" (UniqueName: \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.833431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.833256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.833431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.833380 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.846057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.846020 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8807-6e2730" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" containerName="test-curl-8807-6e2730" containerID="cri-o://31b8625bbb7f1e7a827abb4b3737694422b853e561a9f43c7b699b461e85d770" gracePeriod=2 Oct 30 00:53:25.859178 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:25.859136 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-6e2730\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:25.935569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.935505 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq8t\" (UniqueName: \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.935712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.935570 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.935712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.935621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.936359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.936337 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.937743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.937725 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:25.960471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:25.960451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq8t\" (UniqueName: \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-774nx\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:26.001347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.001323 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:26.032082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.032055 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5"] Oct 30 00:53:26.036931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.036906 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea490f89-c6sw5"] Oct 30 00:53:26.338286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.338144 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx"] Oct 30 00:53:26.858833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.858798 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" event={"ID":"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7","Type":"ContainerStarted","Data":"5a92af42834fddb86c8ef5216d7a874e2fe46d0691113ff9d7723533aab549e4"} Oct 30 00:53:26.859220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.858917 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" event={"ID":"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7","Type":"ContainerStarted","Data":"c126145d3d50c02a2d931acefa93db495875784eab1e74309f2afae7b499554d"} Oct 30 00:53:26.903515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:26.903470 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" podStartSLOduration=1.90345816 podStartE2EDuration="1.90345816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:26.899541929 +0000 UTC m=+1259.139777043" watchObservedRunningTime="2026-10-30 00:53:26.90345816 +0000 UTC m=+1259.143693270" Oct 30 00:53:27.096012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:27.095976 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:27.872617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:27.872585 5938 generic.go:356] "Generic (PLEG): container finished" podID="ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" containerID="5a92af42834fddb86c8ef5216d7a874e2fe46d0691113ff9d7723533aab549e4" exitCode=0 Oct 30 00:53:27.880363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:27.880334 5938 generic.go:356] "Generic (PLEG): container finished" podID="b64f1e92-f473-4f47-aa44-2ea48011a677" containerID="31b8625bbb7f1e7a827abb4b3737694422b853e561a9f43c7b699b461e85d770" exitCode=137 Oct 30 00:53:27.881775 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:27.881752 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37b3109-962c-445c-8120-413dc06d2a60" path="/var/lib/kubelet/pods/b37b3109-962c-445c-8120-413dc06d2a60/volumes" Oct 30 00:53:27.882212 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:27.882188 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" event={"ID":"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7","Type":"ContainerDied","Data":"5a92af42834fddb86c8ef5216d7a874e2fe46d0691113ff9d7723533aab549e4"} Oct 30 00:53:28.256117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.256087 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:28.269112 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:28.269058 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-6e2730\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:28.378574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.378541 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\" (UniqueName: \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\") pod \"b64f1e92-f473-4f47-aa44-2ea48011a677\" (UID: \"b64f1e92-f473-4f47-aa44-2ea48011a677\") " Oct 30 00:53:28.380928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.380898 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb" pod "b64f1e92-f473-4f47-aa44-2ea48011a677" (UID: "b64f1e92-f473-4f47-aa44-2ea48011a677"). InnerVolumeSpecName "kube-api-access-z86cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:28.481076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.481041 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86cb\" (UniqueName: \"kubernetes.io/projected/b64f1e92-f473-4f47-aa44-2ea48011a677-kube-api-access-z86cb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:28.540220 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.540184 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:28.678924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.678818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:28.837673 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:28.837634 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:53:28.872996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.872949 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:28.924586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.924535 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:28.924586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:28.924579 5938 scope.go:122] "RemoveContainer" containerID="31b8625bbb7f1e7a827abb4b3737694422b853e561a9f43c7b699b461e85d770" Oct 30 00:53:28.938324 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:28.938297 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-6e2730\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:28.951284 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:28.951253 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-6e2730\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:29.171015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.170987 5938 scope.go:122] "RemoveContainer" containerID="9dc52395e3d009695a1721ba9d2b593ddcad63343f48da1d0a1ce5f2868ab4d8" Oct 30 00:53:29.184306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.179760 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:29.186151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.186129 5938 scope.go:122] "RemoveContainer" containerID="3fdfb30fd2d40e345b740e7b5e0f39836eef9df80fce3d07ee03d41e79266998" Oct 30 00:53:29.192817 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:29.192781 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-6e2730\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" pod="opendatahub/test-curl-8807-6e2730" Oct 30 00:53:29.196900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.196883 5938 scope.go:122] "RemoveContainer" containerID="f63d9236231717ec73b708a84e2cb045e4eb5f1311ea954e9b4076db57fa1bc9" Oct 30 00:53:29.201405 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.201385 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\") pod \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " Oct 30 00:53:29.201514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.201497 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\") pod \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " Oct 30 00:53:29.201731 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.201713 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\" (UniqueName: \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\") pod \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\" (UID: \"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7\") " Oct 30 00:53:29.201790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.201757 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca" pod "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" (UID: "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:29.202462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.202435 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:29.203481 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.203462 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca" pod "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" (UID: "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:29.203915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.203900 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t" pod "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" (UID: "ae03ffe9-e112-4b5b-93d5-ad053f35e1a7"). InnerVolumeSpecName "kube-api-access-7vq8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:29.209567 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.209552 5938 scope.go:122] "RemoveContainer" containerID="8ccbe316a49b0b6d72a1bbe826bc434ab5b2a9015f6cad196bb31de4821d4693" Oct 30 00:53:29.240782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.240759 5938 scope.go:122] "RemoveContainer" containerID="f2b0cb2cc9adbbc84d214af5780988465557ca9d72e0710adffcefa5bcdf055f" Oct 30 00:53:29.307203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.307169 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:29.307382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.307209 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vq8t\" (UniqueName: \"kubernetes.io/projected/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7-kube-api-access-7vq8t\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:29.883049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.883005 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64f1e92-f473-4f47-aa44-2ea48011a677" path="/var/lib/kubelet/pods/b64f1e92-f473-4f47-aa44-2ea48011a677/volumes" Oct 30 00:53:29.941457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.941431 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" Oct 30 00:53:29.941628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.941475 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx" event={"ID":"ae03ffe9-e112-4b5b-93d5-ad053f35e1a7","Type":"ContainerDied","Data":"c126145d3d50c02a2d931acefa93db495875784eab1e74309f2afae7b499554d"} Oct 30 00:53:29.941628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:29.941533 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c126145d3d50c02a2d931acefa93db495875784eab1e74309f2afae7b499554d" Oct 30 00:53:30.032918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.032889 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk"] Oct 30 00:53:30.037370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.037346 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf145eca-lv4xk"] Oct 30 00:53:30.247147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.247009 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:30.324174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.324141 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:53:30.325369 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.325348 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" containerName="revoke-keys" Oct 30 00:53:30.325433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.325373 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" containerName="revoke-keys" Oct 30 00:53:30.325646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.325632 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" containerName="revoke-keys" Oct 30 00:53:30.328633 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.328615 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.347934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.347910 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:53:30.386012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.385950 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:53:30.389780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.389763 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.398203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.398177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:30.401234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.401186 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bdc256\"" Oct 30 00:53:30.407053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.407028 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bdc256-dockercfg-t6gzr\"" Oct 30 00:53:30.409384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.409363 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:53:30.427555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.427492 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.427669 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.427627 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.427831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.427808 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ztb4\" (UniqueName: \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.427927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.427846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6m5d\" (UniqueName: \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.539971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.531372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.539971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.531449 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.539971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.531598 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ztb4\" (UniqueName: \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.539971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.531641 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6m5d\" (UniqueName: \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.565902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.550768 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.565902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.551488 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.620888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.618599 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ztb4\" (UniqueName: \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\") pod \"payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.672887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.672362 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6m5d\" (UniqueName: \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\") pod \"payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.692891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.680527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:30.718953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.718680 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:30.739888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.726608 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:30.739888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.726694 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:30.954958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.954923 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:30.966973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.966940 5938 generic.go:356] "Generic (PLEG): container finished" podID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerID="d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111" exitCode=0 Oct 30 00:53:30.967213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.967096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerDied","Data":"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111"} Oct 30 00:53:30.967818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.967792 5938 scope.go:122] "RemoveContainer" containerID="d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111" Oct 30 00:53:30.972655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.972620 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:30.972777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.972713 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:30.972777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:30.972769 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:31.038252 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.038212 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42"] Oct 30 00:53:31.046913 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.046067 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d862376-w8c42"] Oct 30 00:53:31.145130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.145095 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:53:31.287848 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.287821 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:53:31.295174 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:31.295135 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d0f7e1_0bf6_4673_92dc_b078cddba4df.slice/crio-53525f0d91fc5943775e7ad628d287e0508cf8af1816fa65f5f7a99a7255bdb6 WatchSource:0}: Error finding container 53525f0d91fc5943775e7ad628d287e0508cf8af1816fa65f5f7a99a7255bdb6: Status 404 returned error can't find the container with id 53525f0d91fc5943775e7ad628d287e0508cf8af1816fa65f5f7a99a7255bdb6 Oct 30 00:53:31.295363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.295337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:31.346113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.345322 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2"] Oct 30 00:53:31.367237 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.367211 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2"] Oct 30 00:53:31.367404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.367302 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.383846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.383823 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:31.384185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.384167 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:53:31.384417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.384401 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:31.384642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.384625 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:31.406495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.406445 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj4zl\" (UniqueName: \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.407281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.407254 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.407451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.407432 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.509221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.509185 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.509221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.509221 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.509536 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.509515 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj4zl\" (UniqueName: \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.509790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.509767 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.511492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.511474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.538755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.538735 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj4zl\" (UniqueName: \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\") pod \"maas-api-revoke-sub-6baa951c-j7qr2\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.693499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.693179 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:31.879305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.879276 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8614d5-d852-428e-9489-748d141cf31f" path="/var/lib/kubelet/pods/0c8614d5-d852-428e-9489-748d141cf31f/volumes" Oct 30 00:53:31.879832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.879816 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c739bb-7dfa-4064-bbfd-826e0ebd4e16" path="/var/lib/kubelet/pods/97c739bb-7dfa-4064-bbfd-826e0ebd4e16/volumes" Oct 30 00:53:31.978661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.978567 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" event={"ID":"bba98624-b644-4722-894c-03259b6ed99c","Type":"ContainerStarted","Data":"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23"} Oct 30 00:53:31.978661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.978635 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" event={"ID":"bba98624-b644-4722-894c-03259b6ed99c","Type":"ContainerStarted","Data":"db735688a396d2b0e731c32d3b527f6cd107c06ce22e2abdd907725590d2c8e9"} Oct 30 00:53:31.979166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.978667 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:31.980479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.980437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" event={"ID":"d4d0f7e1-0bf6-4673-92dc-b078cddba4df","Type":"ContainerStarted","Data":"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059"} Oct 30 00:53:31.980631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.980494 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" event={"ID":"d4d0f7e1-0bf6-4673-92dc-b078cddba4df","Type":"ContainerStarted","Data":"53525f0d91fc5943775e7ad628d287e0508cf8af1816fa65f5f7a99a7255bdb6"} Oct 30 00:53:31.980631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.980549 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:31.982644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:31.982623 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerStarted","Data":"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9"} Oct 30 00:53:32.016857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.016826 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2"] Oct 30 00:53:32.017453 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:32.017428 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d0e260_0cf8_4f20_acfc_b73be720cccd.slice/crio-f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e WatchSource:0}: Error finding container f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e: Status 404 returned error can't find the container with id f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e Oct 30 00:53:32.038307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.038267 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" podStartSLOduration=2.038251866 podStartE2EDuration="2.038251866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:32.034050562 +0000 UTC m=+1264.274285677" watchObservedRunningTime="2026-10-30 00:53:32.038251866 +0000 UTC m=+1264.278486981" Oct 30 00:53:32.122544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.122491 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" podStartSLOduration=2.12247583 podStartE2EDuration="2.12247583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:32.120556004 +0000 UTC m=+1264.360791150" watchObservedRunningTime="2026-10-30 00:53:32.12247583 +0000 UTC m=+1264.362710943" Oct 30 00:53:32.426162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.426136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:32.572156 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.572121 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:32.789939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.783727 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:32.852884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.847240 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:53:32.852884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.851273 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:32.852884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:32.851326 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:33.046949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:33.046395 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" event={"ID":"84d0e260-0cf8-4f20-acfc-b73be720cccd","Type":"ContainerStarted","Data":"44b84f6e8b035945b159b5de65be68ba80ddd055c3ef45373aa23612034c81b6"} Oct 30 00:53:33.046949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:33.046448 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" event={"ID":"84d0e260-0cf8-4f20-acfc-b73be720cccd","Type":"ContainerStarted","Data":"f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e"} Oct 30 00:53:33.133889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:33.133103 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" podStartSLOduration=2.133039046 podStartE2EDuration="2.133039046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:33.094147043 +0000 UTC m=+1265.334382158" watchObservedRunningTime="2026-10-30 00:53:33.133039046 +0000 UTC m=+1265.373274158" Oct 30 00:53:33.845894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:33.839292 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:33.845894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:33.839354 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:34.058883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:34.058079 5938 generic.go:356] "Generic (PLEG): container finished" podID="84d0e260-0cf8-4f20-acfc-b73be720cccd" containerID="44b84f6e8b035945b159b5de65be68ba80ddd055c3ef45373aa23612034c81b6" exitCode=0 Oct 30 00:53:34.058883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:34.058124 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" event={"ID":"84d0e260-0cf8-4f20-acfc-b73be720cccd","Type":"ContainerDied","Data":"44b84f6e8b035945b159b5de65be68ba80ddd055c3ef45373aa23612034c81b6"} Oct 30 00:53:34.674887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:34.671616 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:34.838397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:34.838357 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:34.838661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:34.838632 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:35.138882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.131918 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4"] Oct 30 00:53:35.178895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.174650 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae9b907-mnpv4"] Oct 30 00:53:35.842936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.839542 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:35.842936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.839603 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:35.867887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.867124 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:35.929888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:35.929665 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 00:53:36.018889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.017954 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.092901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.091358 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.092901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.092091 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.275286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.238068 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 00:53:36.275286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.238675 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 00:53:36.305497 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.305463 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89beefc6-8952-4c5b-84c5-81142d45353a" path="/var/lib/kubelet/pods/89beefc6-8952-4c5b-84c5-81142d45353a/volumes" Oct 30 00:53:36.308061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.306257 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:53:36.351519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.351483 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.365244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.364220 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.450533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.404206 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.450933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.450892 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:36.676889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.674331 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:36.773919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.760089 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:36.831302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.831220 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:36.837503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.837463 5938 patch_prober.go:29] interesting pod/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 00:53:36.837621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.837519 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 00:53:36.881017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.880971 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 00:53:36.882595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.882372 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\" (UniqueName: \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\") pod \"84d0e260-0cf8-4f20-acfc-b73be720cccd\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " Oct 30 00:53:36.882595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.882519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\") pod \"84d0e260-0cf8-4f20-acfc-b73be720cccd\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " Oct 30 00:53:36.882797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.882730 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\") pod \"84d0e260-0cf8-4f20-acfc-b73be720cccd\" (UID: \"84d0e260-0cf8-4f20-acfc-b73be720cccd\") " Oct 30 00:53:36.883435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.883408 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca" pod "84d0e260-0cf8-4f20-acfc-b73be720cccd" (UID: "84d0e260-0cf8-4f20-acfc-b73be720cccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:36.932892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.924470 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca" pod "84d0e260-0cf8-4f20-acfc-b73be720cccd" (UID: "84d0e260-0cf8-4f20-acfc-b73be720cccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:36.932892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:36.927258 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl" pod "84d0e260-0cf8-4f20-acfc-b73be720cccd" (UID: "84d0e260-0cf8-4f20-acfc-b73be720cccd"). InnerVolumeSpecName "kube-api-access-pj4zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:37.063034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.052202 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.063034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.052241 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d0e260-0cf8-4f20-acfc-b73be720cccd-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.063034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.052255 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj4zl\" (UniqueName: \"kubernetes.io/projected/84d0e260-0cf8-4f20-acfc-b73be720cccd-kube-api-access-pj4zl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:37.093044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.092944 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.094266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.094241 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:53:37.322908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.322117 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.373351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.373313 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:37.374279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.374259 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:53:37.403885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.383324 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 00:53:37.440490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.440463 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 00:53:37.753688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.753603 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" Oct 30 00:53:37.753688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.753633 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2" event={"ID":"84d0e260-0cf8-4f20-acfc-b73be720cccd","Type":"ContainerDied","Data":"f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e"} Oct 30 00:53:37.753999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.753688 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f418a1d1b8bc80f8b8adf40ffdf42fa264346d3043b98f4a6e42991521fa050e" Oct 30 00:53:37.925615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.925582 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:53:37.926022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.925988 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:53:37.926898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:37.926843 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:53:38.172908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:38.172836 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:39.818115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:39.816672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:41.029144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.029108 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6"] Oct 30 00:53:41.040905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.040873 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceadfc08-zscr6"] Oct 30 00:53:41.298404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.298303 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:41.316128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.311119 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" containerID="cri-o://7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643" gracePeriod=25 Oct 30 00:53:41.746933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.746902 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8"] Oct 30 00:53:41.747595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.747582 5938 cpu_manager.go:414] "containerMap: removing container" podUID="84d0e260-0cf8-4f20-acfc-b73be720cccd" containerName="revoke-keys" Oct 30 00:53:41.747641 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.747596 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d0e260-0cf8-4f20-acfc-b73be720cccd" containerName="revoke-keys" Oct 30 00:53:41.747751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.747742 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d0e260-0cf8-4f20-acfc-b73be720cccd" containerName="revoke-keys" Oct 30 00:53:41.750392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.750377 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.774558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.774532 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:53:41.776181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.776154 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:53:41.776387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.776371 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:53:41.779599 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.779578 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:53:41.784937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.784911 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8"] Oct 30 00:53:41.876353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.876310 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwjq\" (UniqueName: \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.876660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.876639 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.876718 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.876692 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.878007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.877987 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b437c0b7-95ff-486b-bec0-f07d6d73b792" path="/var/lib/kubelet/pods/b437c0b7-95ff-486b-bec0-f07d6d73b792/volumes" Oct 30 00:53:41.978517 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.978492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.978611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.978528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.978611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.978578 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwjq\" (UniqueName: \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.979211 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.979190 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:41.981044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:41.981023 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:42.001150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.001100 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwjq\" (UniqueName: \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\") pod \"maas-api-revoke-sub-1d9aaf92-tdgk8\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:42.026402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.026379 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:42.063684 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.063660 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:42.374352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.374322 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8"] Oct 30 00:53:42.374776 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:42.374745 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe70a5ca_16a8_48ee_987d_45e2c5fe236c.slice/crio-86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b WatchSource:0}: Error finding container 86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b: Status 404 returned error can't find the container with id 86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b Oct 30 00:53:42.803081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.803052 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" event={"ID":"be70a5ca-16a8-48ee-987d-45e2c5fe236c","Type":"ContainerStarted","Data":"445f6d742f02c2529d74198abce0fe4001f87bc1d23d8bca31fd3031199930c1"} Oct 30 00:53:42.803222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.803104 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" event={"ID":"be70a5ca-16a8-48ee-987d-45e2c5fe236c","Type":"ContainerStarted","Data":"86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b"} Oct 30 00:53:42.852050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.852009 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" podStartSLOduration=1.851996838 podStartE2EDuration="1.851996838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:42.842966656 +0000 UTC m=+1275.083201771" watchObservedRunningTime="2026-10-30 00:53:42.851996838 +0000 UTC m=+1275.092231982" Oct 30 00:53:42.949975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:42.949837 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:43.052401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:43.051291 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:53:43.052401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:43.051357 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:53:43.814126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:43.814090 5938 generic.go:356] "Generic (PLEG): container finished" podID="be70a5ca-16a8-48ee-987d-45e2c5fe236c" containerID="445f6d742f02c2529d74198abce0fe4001f87bc1d23d8bca31fd3031199930c1" exitCode=0 Oct 30 00:53:43.814532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:43.814172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" event={"ID":"be70a5ca-16a8-48ee-987d-45e2c5fe236c","Type":"ContainerDied","Data":"445f6d742f02c2529d74198abce0fe4001f87bc1d23d8bca31fd3031199930c1"} Oct 30 00:53:44.029286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:44.029257 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk"] Oct 30 00:53:44.033640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:44.033615 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b99671f-bwlxk"] Oct 30 00:53:44.455055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:44.455024 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:44.537023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:44.536976 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:53:44.996525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:44.996499 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:45.126075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.126047 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\") pod \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " Oct 30 00:53:45.126231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.126196 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\") pod \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " Oct 30 00:53:45.126280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.126241 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\" (UniqueName: \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\") pod \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\" (UID: \"be70a5ca-16a8-48ee-987d-45e2c5fe236c\") " Oct 30 00:53:45.126396 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.126373 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca" pod "be70a5ca-16a8-48ee-987d-45e2c5fe236c" (UID: "be70a5ca-16a8-48ee-987d-45e2c5fe236c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:45.126897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.126831 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:45.128240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.128217 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca" pod "be70a5ca-16a8-48ee-987d-45e2c5fe236c" (UID: "be70a5ca-16a8-48ee-987d-45e2c5fe236c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:45.128388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.128368 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq" pod "be70a5ca-16a8-48ee-987d-45e2c5fe236c" (UID: "be70a5ca-16a8-48ee-987d-45e2c5fe236c"). InnerVolumeSpecName "kube-api-access-ttwjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:45.228747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.228717 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be70a5ca-16a8-48ee-987d-45e2c5fe236c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:45.228747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.228748 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttwjq\" (UniqueName: \"kubernetes.io/projected/be70a5ca-16a8-48ee-987d-45e2c5fe236c-kube-api-access-ttwjq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:45.773294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.773253 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw"] Oct 30 00:53:45.774094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.774070 5938 cpu_manager.go:414] "containerMap: removing container" podUID="be70a5ca-16a8-48ee-987d-45e2c5fe236c" containerName="revoke-keys" Oct 30 00:53:45.774094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.774092 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be70a5ca-16a8-48ee-987d-45e2c5fe236c" containerName="revoke-keys" Oct 30 00:53:45.774321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.774306 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="be70a5ca-16a8-48ee-987d-45e2c5fe236c" containerName="revoke-keys" Oct 30 00:53:45.778000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.777973 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:45.796406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.796381 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw"] Oct 30 00:53:45.834471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.834445 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" Oct 30 00:53:45.834471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.834455 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8" event={"ID":"be70a5ca-16a8-48ee-987d-45e2c5fe236c","Type":"ContainerDied","Data":"86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b"} Oct 30 00:53:45.834760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.834506 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a98d471769de8f36c25f19e99bc27da041223f7998c1d881f789dcb9a9649b" Oct 30 00:53:45.879853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.879828 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3424d7-a7f1-41c1-9c3e-80d2580fe041" path="/var/lib/kubelet/pods/2e3424d7-a7f1-41c1-9c3e-80d2580fe041/volumes" Oct 30 00:53:45.936781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.936755 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:45.936925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.936818 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:45.937099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:45.937079 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.029953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.029892 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg"] Oct 30 00:53:46.033910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.033889 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5dca60-4m4gg"] Oct 30 00:53:46.038384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.038366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.038474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.038424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.038916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.038667 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.039420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.039398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.041052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.041034 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.060603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.060585 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\") pod \"maas-api-revoke-sub-4739c7bc-ptrfw\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.079322 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.079295 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:46.093767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.093674 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:46.457772 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:53:46.457744 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946f2cc2_9911_428e_af5d_4e0492b44370.slice/crio-eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2 WatchSource:0}: Error finding container eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2: Status 404 returned error can't find the container with id eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2 Oct 30 00:53:46.458387 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.458366 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw"] Oct 30 00:53:46.845208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.845177 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" event={"ID":"946f2cc2-9911-428e-af5d-4e0492b44370","Type":"ContainerStarted","Data":"bf4e6c7eaeaf6ade6782071ad2f2a356634c8de825fed74cdc6c14bbfffa52af"} Oct 30 00:53:46.845329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.845231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" event={"ID":"946f2cc2-9911-428e-af5d-4e0492b44370","Type":"ContainerStarted","Data":"eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2"} Oct 30 00:53:46.888324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:46.888280 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" podStartSLOduration=1.8882679549999999 podStartE2EDuration="1.888267955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:46.883473106 +0000 UTC m=+1279.123708218" watchObservedRunningTime="2026-10-30 00:53:46.888267955 +0000 UTC m=+1279.128503068" Oct 30 00:53:47.855505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:47.855477 5938 generic.go:356] "Generic (PLEG): container finished" podID="946f2cc2-9911-428e-af5d-4e0492b44370" containerID="bf4e6c7eaeaf6ade6782071ad2f2a356634c8de825fed74cdc6c14bbfffa52af" exitCode=0 Oct 30 00:53:47.855917 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:47.855563 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" event={"ID":"946f2cc2-9911-428e-af5d-4e0492b44370","Type":"ContainerDied","Data":"bf4e6c7eaeaf6ade6782071ad2f2a356634c8de825fed74cdc6c14bbfffa52af"} Oct 30 00:53:47.863709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:47.863689 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:47.895592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:47.895485 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646635fd-4d84-42a1-972e-71312c047d3c" path="/var/lib/kubelet/pods/646635fd-4d84-42a1-972e-71312c047d3c/volumes" Oct 30 00:53:48.677892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:48.677835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:49.038803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.038782 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:49.087634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.087605 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\") pod \"946f2cc2-9911-428e-af5d-4e0492b44370\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " Oct 30 00:53:49.087764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.087654 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\") pod \"946f2cc2-9911-428e-af5d-4e0492b44370\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " Oct 30 00:53:49.087847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.087828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\") pod \"946f2cc2-9911-428e-af5d-4e0492b44370\" (UID: \"946f2cc2-9911-428e-af5d-4e0492b44370\") " Oct 30 00:53:49.088069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.088042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca" pod "946f2cc2-9911-428e-af5d-4e0492b44370" (UID: "946f2cc2-9911-428e-af5d-4e0492b44370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:53:49.088524 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.088500 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:49.090905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.090854 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca" pod "946f2cc2-9911-428e-af5d-4e0492b44370" (UID: "946f2cc2-9911-428e-af5d-4e0492b44370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:53:49.117062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.117038 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9" pod "946f2cc2-9911-428e-af5d-4e0492b44370" (UID: "946f2cc2-9911-428e-af5d-4e0492b44370"). InnerVolumeSpecName "kube-api-access-qc6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:53:49.190294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.190270 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6z9\" (UniqueName: \"kubernetes.io/projected/946f2cc2-9911-428e-af5d-4e0492b44370-kube-api-access-qc6z9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:49.190294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.190293 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/946f2cc2-9911-428e-af5d-4e0492b44370-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:53:49.457007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.456966 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:49.874587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.874562 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" Oct 30 00:53:49.878751 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.878725 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw" event={"ID":"946f2cc2-9911-428e-af5d-4e0492b44370","Type":"ContainerDied","Data":"eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2"} Oct 30 00:53:49.878918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:49.878764 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac75e53e2ca34c791b4af07daff38adaf807de85d6ca21443ca564e0f73eea2" Oct 30 00:53:50.976243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:50.976209 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:53.045882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.041471 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q"] Oct 30 00:53:53.054254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.053804 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948ad435-dhv4q"] Oct 30 00:53:53.137209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.137156 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:53.327955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.327782 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:53.501915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.494591 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:53.919890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.913569 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:53.919890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:53.913744 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b84395-88a9-4887-b9ce-5e67e258f1a9" path="/var/lib/kubelet/pods/58b84395-88a9-4887-b9ce-5e67e258f1a9/volumes" Oct 30 00:53:54.526894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.515917 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:53:54.526894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.517134 5938 cpu_manager.go:414] "containerMap: removing container" podUID="946f2cc2-9911-428e-af5d-4e0492b44370" containerName="revoke-keys" Oct 30 00:53:54.526894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.517155 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946f2cc2-9911-428e-af5d-4e0492b44370" containerName="revoke-keys" Oct 30 00:53:54.526894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.517374 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="946f2cc2-9911-428e-af5d-4e0492b44370" containerName="revoke-keys" Oct 30 00:53:54.538286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.538255 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:54.582069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.582019 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 00:53:54.760149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.760093 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:53:54.760534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.760503 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781194 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmdm\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781244 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781300 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781349 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781427 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781519 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781558 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.781640 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:54.858153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.812720 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-52785f5e-openshift-default-dockercfg-q7mb4\"" Oct 30 00:53:54.915983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:54.915939 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.001809 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.001771 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.003622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.002667 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.004010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.003984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.010262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.010233 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.010668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.010448 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.010901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.010882 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.011421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.011206 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.011655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.011636 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmdm\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.013362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.013338 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.017331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.017306 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.017940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.017916 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.018475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.018453 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.019697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.019669 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.020038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.020012 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.125939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.104621 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": context deadline exceeded" Oct 30 00:53:55.151593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.151559 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.152563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.152522 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.202059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.184482 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.202337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.185123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:55.296888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.248233 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmdm\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.296888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.282465 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d" podUID="5826cc58-3128-42cc-b1e6-06847d5ed3b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.339895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.339436 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.372043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.359418 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\") pod \"e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.419121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.419029 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": context deadline exceeded" Oct 30 00:53:55.446926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.444440 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.609455 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.609424 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:55.688169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.688074 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 00:53:55.698163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.698132 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 00:53:55.766236 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.766191 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:55.959043 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.938944 5938 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:55.960700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:55.960661 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:53:56.240971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.230959 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:53:56.269215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.269048 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 00:53:56.774744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.774714 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:53:56.782186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.781982 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:56.782186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.782072 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:56.782186 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:56.782133 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:53:57.605044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:57.604243 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:57.640153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:57.632606 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerStarted","Data":"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f"} Oct 30 00:53:57.644009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:57.643970 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerStarted","Data":"8f2ca84b86d3017e139eafaa97f6f213d3fbcdc57746ad95253f9f57113cbf75"} Oct 30 00:53:57.734086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:57.734019 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podStartSLOduration=3.734000866 podStartE2EDuration="3.734000866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:57.698504502 +0000 UTC m=+1289.938739640" watchObservedRunningTime="2026-10-30 00:53:57.734000866 +0000 UTC m=+1289.974235979" Oct 30 00:53:58.610944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:58.610903 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:58.616467 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:58.616446 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:58.641911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:58.641888 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:58.642753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:58.642734 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:53:58.839386 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:53:58.839335 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:53:59.295238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:53:59.295164 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:00.189127 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:00.189094 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:00.331762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:00.331729 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:01.139930 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.139893 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:01.657166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.657137 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:54:01.678832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.678794 5938 generic.go:356] "Generic (PLEG): container finished" podID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerID="7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643" exitCode=0 Oct 30 00:54:01.679059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.678845 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" Oct 30 00:54:01.679059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.678846 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" event={"ID":"6d8f658f-e268-474b-87dd-2a0576f7c577","Type":"ContainerDied","Data":"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643"} Oct 30 00:54:01.679059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.678938 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn" event={"ID":"6d8f658f-e268-474b-87dd-2a0576f7c577","Type":"ContainerDied","Data":"45e52f1e6232de84d4f7d4eb542bdda9fa4aff7b85a9e416b66b3c72a0b0fb77"} Oct 30 00:54:01.679059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.678971 5938 scope.go:122] "RemoveContainer" containerID="7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643" Oct 30 00:54:01.692977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.692947 5938 scope.go:122] "RemoveContainer" containerID="7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643" Oct 30 00:54:01.693289 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:01.693261 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643\": container with ID starting with 7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643 not found: ID does not exist" containerID="7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643" Oct 30 00:54:01.693365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.693302 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643"} err="failed to get container status \"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643\": rpc error: code = NotFound desc = could not find container \"7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643\": container with ID starting with 7fe11baf00465eb9adea8f19662bcfbbdd782d2eaac4331587967422040ea643 not found: ID does not exist" Oct 30 00:54:01.743570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.743498 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\") pod \"6d8f658f-e268-474b-87dd-2a0576f7c577\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " Oct 30 00:54:01.743768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.743749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\" (UniqueName: \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\") pod \"6d8f658f-e268-474b-87dd-2a0576f7c577\" (UID: \"6d8f658f-e268-474b-87dd-2a0576f7c577\") " Oct 30 00:54:01.744691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.744664 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume" pod "6d8f658f-e268-474b-87dd-2a0576f7c577" (UID: "6d8f658f-e268-474b-87dd-2a0576f7c577"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:01.758748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.758719 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9" pod "6d8f658f-e268-474b-87dd-2a0576f7c577" (UID: "6d8f658f-e268-474b-87dd-2a0576f7c577"). InnerVolumeSpecName "kube-api-access-djcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:01.846269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.846242 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8f658f-e268-474b-87dd-2a0576f7c577-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:01.846269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:01.846271 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcm9\" (UniqueName: \"kubernetes.io/projected/6d8f658f-e268-474b-87dd-2a0576f7c577-kube-api-access-djcm9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:02.020347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:02.020283 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:54:02.032058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:02.032030 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-khvkn"] Oct 30 00:54:02.972342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:02.972307 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:03.061108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.061079 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:54:03.061829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.061814 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" Oct 30 00:54:03.061893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.061832 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" Oct 30 00:54:03.062022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.062012 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" containerName="payload-processing" Oct 30 00:54:03.065934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.065916 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.085708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.085681 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:54:03.090577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.090554 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-52785f5e\"" Oct 30 00:54:03.121412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.121388 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:03.132668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.132634 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:54:03.138925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.138904 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.153207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.153175 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-52785f5e-dockercfg-p86fl\"" Oct 30 00:54:03.153962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.153935 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:54:03.165907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.165880 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.166069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.165931 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv554\" (UniqueName: \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.317942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.295507 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.317942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.295562 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv554\" (UniqueName: \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.317942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.295602 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd2p\" (UniqueName: \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.362908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.341797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.362908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.356662 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:03.389980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.367973 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.471749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.445453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd2p\" (UniqueName: \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.471749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.445878 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.471749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.447194 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.574972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.571726 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd2p\" (UniqueName: \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\") pod \"payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.583678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.583607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv554\" (UniqueName: \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\") pod \"payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.681709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.681678 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:03.765992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.765961 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:03.885061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:03.881118 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8f658f-e268-474b-87dd-2a0576f7c577" path="/var/lib/kubelet/pods/6d8f658f-e268-474b-87dd-2a0576f7c577/volumes" Oct 30 00:54:04.014085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.014055 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:54:04.014540 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:04.014229 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181fdbda_e2e0_4639_84c1_fe7d2113131d.slice/crio-d498c9ee62caea96b7a36adafff24a147346683d40044e4ab2d0832dbc5e5449 WatchSource:0}: Error finding container d498c9ee62caea96b7a36adafff24a147346683d40044e4ab2d0832dbc5e5449: Status 404 returned error can't find the container with id d498c9ee62caea96b7a36adafff24a147346683d40044e4ab2d0832dbc5e5449 Oct 30 00:54:04.110901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.110855 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:54:04.111824 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:04.111798 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa1136a_ea8d_42ec_bbcd_c777768d6c6c.slice/crio-4fea11969b104412eb34d213a531a95fac0206e7674b8f15a59e1e75260c266b WatchSource:0}: Error finding container 4fea11969b104412eb34d213a531a95fac0206e7674b8f15a59e1e75260c266b: Status 404 returned error can't find the container with id 4fea11969b104412eb34d213a531a95fac0206e7674b8f15a59e1e75260c266b Oct 30 00:54:04.250731 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.250705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:04.722295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.722258 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" event={"ID":"181fdbda-e2e0-4639-84c1-fe7d2113131d","Type":"ContainerStarted","Data":"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652"} Oct 30 00:54:04.722471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.722363 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:04.722471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.722392 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" event={"ID":"181fdbda-e2e0-4639-84c1-fe7d2113131d","Type":"ContainerStarted","Data":"d498c9ee62caea96b7a36adafff24a147346683d40044e4ab2d0832dbc5e5449"} Oct 30 00:54:04.724084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.724059 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" event={"ID":"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c","Type":"ContainerStarted","Data":"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9"} Oct 30 00:54:04.724177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.724104 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" event={"ID":"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c","Type":"ContainerStarted","Data":"4fea11969b104412eb34d213a531a95fac0206e7674b8f15a59e1e75260c266b"} Oct 30 00:54:04.724227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.724195 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:04.761914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.761833 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" podStartSLOduration=1.761820927 podStartE2EDuration="1.761820927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:04.760804217 +0000 UTC m=+1297.001039332" watchObservedRunningTime="2026-10-30 00:54:04.761820927 +0000 UTC m=+1297.002056040" Oct 30 00:54:04.801463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.801422 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" podStartSLOduration=1.801410358 podStartE2EDuration="1.801410358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:04.79885425 +0000 UTC m=+1297.039089367" watchObservedRunningTime="2026-10-30 00:54:04.801410358 +0000 UTC m=+1297.041645471" Oct 30 00:54:04.942972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:04.942843 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.086320 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.086281 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.432368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.432332 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.574009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.571944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:05.776325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.776233 5938 generic.go:356] "Generic (PLEG): container finished" podID="e369062e-6423-4537-91fa-55edba20c3af" containerID="921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f" exitCode=0 Oct 30 00:54:05.777684 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.776534 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerDied","Data":"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f"} Oct 30 00:54:05.779172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.779153 5938 scope.go:122] "RemoveContainer" containerID="921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f" Oct 30 00:54:05.782139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.782106 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:05.782327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.782309 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:05.782449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:05.782434 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:06.705829 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:06.705795 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:06.792521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:06.792478 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerStarted","Data":"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb"} Oct 30 00:54:07.508887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:07.508226 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:54:07.510486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:07.510456 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:07.510623 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:07.510523 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:07.524768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:07.524746 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:54:08.511113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:08.509250 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:08.511113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:08.509305 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:08.541135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:08.540668 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:09.511562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:09.511525 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:09.512306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:09.512249 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:09.525317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:09.525293 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:09.669604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:09.669570 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:10.082613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:10.082579 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85"] Oct 30 00:54:10.124134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:10.124100 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec098887-wcm85"] Oct 30 00:54:10.407675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:10.407636 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:10.508347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:10.508284 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:10.508667 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:10.508635 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:11.215452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:11.213961 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:11.365374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:11.365343 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:11.565938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:11.565663 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:11.565938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:11.565761 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:11.881551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:11.881478 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64da4cf-20ca-401d-8861-6e8b8b9341cf" path="/var/lib/kubelet/pods/e64da4cf-20ca-401d-8861-6e8b8b9341cf/volumes" Oct 30 00:54:12.312956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:12.308659 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:12.507688 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:12.507648 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 00:54:12.507941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:12.507701 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 00:54:13.511628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:13.511601 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:54:13.512937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:13.512916 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:54:14.364911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:14.364850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:15.781002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:15.780974 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:54:15.781382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:15.781036 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:54:16.526292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:16.526258 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:18.030355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:18.030322 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46"] Oct 30 00:54:18.041573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:18.041545 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dbe2ff0-nlc46"] Oct 30 00:54:18.081010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:18.080980 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:19.461577 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:19.461545 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:19.882924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:19.882845 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7acdeb8-6605-4304-bc5b-40f4125e9114" path="/var/lib/kubelet/pods/a7acdeb8-6605-4304-bc5b-40f4125e9114/volumes" Oct 30 00:54:20.299687 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:20.299606 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:20.444413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:20.444381 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx"] Oct 30 00:54:20.455145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:20.455118 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-774nx"] Oct 30 00:54:20.508231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:20.508202 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:20.648572 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:20.648532 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:21.878833 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:21.878800 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae03ffe9-e112-4b5b-93d5-ad053f35e1a7" path="/var/lib/kubelet/pods/ae03ffe9-e112-4b5b-93d5-ad053f35e1a7/volumes" Oct 30 00:54:22.425384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:22.425357 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:24.117720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:24.117677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:24.758562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:24.758527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:24.903198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:24.903159 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:25.936394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:25.936354 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:27.028934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:27.028904 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl"] Oct 30 00:54:27.039892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:27.039853 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9e216c-vk7dl"] Oct 30 00:54:27.727353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:27.727316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:27.877908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:27.877831 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:27.882055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:27.882024 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaee8520-fb23-4258-9ff2-a2e1510eb145" path="/var/lib/kubelet/pods/aaee8520-fb23-4258-9ff2-a2e1510eb145/volumes" Oct 30 00:54:28.111431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:28.111399 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:28.354565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:28.354536 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:28.836727 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:28.836692 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:54:29.140646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.140614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:29.263172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.263138 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:54:29.277393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.277373 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.287056 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.287028 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:29.288581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.288552 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:54:29.291527 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.291502 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d7a6db25\"" Oct 30 00:54:29.291637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.291581 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d7a6db25-dockercfg-4fgll\"" Oct 30 00:54:29.490982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.470349 5938 scope.go:122] "RemoveContainer" containerID="c57a9da34d02b3fd655e090bca6606c6113fc503a51b346b29ace3e7d438736f" Oct 30 00:54:29.521040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.520970 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkpn\" (UniqueName: \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.521549 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.521523 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.626291 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.626243 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkpn\" (UniqueName: \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.626707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.626674 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.628327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.628096 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.663319 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.663291 5938 scope.go:122] "RemoveContainer" containerID="5c41ad629738a9f58e0231d1ba408e40eba1f0bedd4dacc740cd41c177ac7af2" Oct 30 00:54:29.682842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.682808 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkpn\" (UniqueName: \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:29.709822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.709776 5938 scope.go:122] "RemoveContainer" containerID="d5219b6404a023b1658a397e58ea603c16fbf70aae3523e12ecc4680864108b9" Oct 30 00:54:29.720839 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.720815 5938 scope.go:122] "RemoveContainer" containerID="9f725dc337c42a624e998494a3e76cb7799801693789f8a915d6ae8b2a8d9604" Oct 30 00:54:29.733856 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.733836 5938 scope.go:122] "RemoveContainer" containerID="f44bc9a1dd147f2735defad66a413ae4ec376e35bf25b7e851e694002833c287" Oct 30 00:54:29.747112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.747091 5938 scope.go:122] "RemoveContainer" containerID="44a2cd248d81c82ba6da4c7e13a174280b4dbf3994ff1223ae709aa51e6ebe7f" Oct 30 00:54:29.757772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.757746 5938 scope.go:122] "RemoveContainer" containerID="af44c5fb3d1e04d35dd31bca2c0e0582e7148071403a6c9adb7c067c2f459119" Oct 30 00:54:29.769761 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.769652 5938 scope.go:122] "RemoveContainer" containerID="9db9494a210cb2e7dfbaa20ae66209408996b1e6d8b3bf47aee39d62356c0aba" Oct 30 00:54:29.781355 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.781330 5938 scope.go:122] "RemoveContainer" containerID="e02756a7dda3b34ca0bb759acc60b5cf79bcf4d14997f880d1e87eeae0ca7c28" Oct 30 00:54:29.791548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.791508 5938 scope.go:122] "RemoveContainer" containerID="0e671fd6632b0759cc7b4b86195d9b036dfd3d77a467912bbb91ffe61208bbe8" Oct 30 00:54:29.906709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:29.906675 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:30.210228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:30.210039 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:54:30.232073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:30.219011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:30.290178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:30.287370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" event={"ID":"d3f9d5cb-58df-4822-8ce1-cd506007c9fb","Type":"ContainerStarted","Data":"6a5f72951882b2156cb38b7530c55cfd53950ab699d5f914ff70b7266aeb4a50"} Oct 30 00:54:31.299288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:31.299252 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" event={"ID":"d3f9d5cb-58df-4822-8ce1-cd506007c9fb","Type":"ContainerStarted","Data":"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc"} Oct 30 00:54:31.299812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:31.299347 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:31.341937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:31.341887 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" podStartSLOduration=2.341858678 podStartE2EDuration="2.341858678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:31.337943584 +0000 UTC m=+1323.578178700" watchObservedRunningTime="2026-10-30 00:54:31.341858678 +0000 UTC m=+1323.582093788" Oct 30 00:54:31.941111 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:31.941080 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:33.582050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:33.582002 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:35.077797 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:35.077766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:36.029129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.029099 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt"] Oct 30 00:54:36.040182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.040133 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4ba46-sgmzt"] Oct 30 00:54:36.467001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.466971 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:54:36.471724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.471700 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.505346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.505317 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:54:36.642016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.641989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642050 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642193 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642235 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642298 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbbw\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642374 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642419 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.642630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.642618 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.704891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.701625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.743761 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.743828 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.743927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744007 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744041 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744074 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbbw\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744107 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744201 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744444 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744711 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.744780 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.745011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.744986 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.745112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.745009 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.746928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.746903 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.747055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.746903 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.766618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.766589 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.767020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.767003 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbbw\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.787099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.787072 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:36.960456 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:36.960420 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:37.117409 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:37.117379 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2181c326_e2df_45bc_b795_9049cafad0fe.slice/crio-e22657ac981b7754df32e197dab01cf5450e92a5a3f07968d69b4fb459225f15 WatchSource:0}: Error finding container e22657ac981b7754df32e197dab01cf5450e92a5a3f07968d69b4fb459225f15: Status 404 returned error can't find the container with id e22657ac981b7754df32e197dab01cf5450e92a5a3f07968d69b4fb459225f15 Oct 30 00:54:37.121627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.121594 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:54:37.121828 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.121785 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:37.121964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.121939 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:37.122053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.122030 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:54:37.345341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.345309 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:54:37.345573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.345540 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" containerID="cri-o://16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a" gracePeriod=30 Oct 30 00:54:37.356021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.356000 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:54:37.356392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.356349 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" containerID="cri-o://02594ba4bb29dce5d94014c888769ab56cc4fff359d51177fd75ad6eee661860" gracePeriod=30 Oct 30 00:54:37.362933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.362793 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:54:37.363054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.363013 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" containerID="cri-o://b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78" gracePeriod=30 Oct 30 00:54:37.369360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.369333 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:54:37.369601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.369556 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" containerID="cri-o://0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325" gracePeriod=30 Oct 30 00:54:37.391371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.391344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" event={"ID":"2181c326-e2df-45bc-b795-9049cafad0fe","Type":"ContainerStarted","Data":"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82"} Oct 30 00:54:37.396937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.396188 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" event={"ID":"2181c326-e2df-45bc-b795-9049cafad0fe","Type":"ContainerStarted","Data":"e22657ac981b7754df32e197dab01cf5450e92a5a3f07968d69b4fb459225f15"} Oct 30 00:54:37.444660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.443844 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podStartSLOduration=1.443827146 podStartE2EDuration="1.443827146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:37.439457236 +0000 UTC m=+1329.679692351" watchObservedRunningTime="2026-10-30 00:54:37.443827146 +0000 UTC m=+1329.684062260" Oct 30 00:54:37.787501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.787420 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:37.908234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.908195 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c79c72e-900b-4d41-9fc7-ef8c82e9fee3" path="/var/lib/kubelet/pods/5c79c72e-900b-4d41-9fc7-ef8c82e9fee3/volumes" Oct 30 00:54:37.965001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:37.964966 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:38.129351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:38.129310 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:38.685911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:38.677561 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:38.861075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:38.852615 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:38.861075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:38.852664 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:38.861075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:38.853396 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:39.377479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:39.377446 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:39.809357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:39.802210 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:39.809357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:39.802270 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:40.148802 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:40.148770 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:40.642158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:40.642114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:40.785887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:40.785654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:40.800883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:40.789211 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:40.800883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:40.789261 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:41.302204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:41.292718 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:41.806886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:41.806409 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:41.806886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:41.806469 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:42.323883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.320031 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:54:42.434281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.431448 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:42.471280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.471092 5938 generic.go:356] "Generic (PLEG): container finished" podID="891703a9-fa46-4691-89ec-7a7777c31f45" containerID="02594ba4bb29dce5d94014c888769ab56cc4fff359d51177fd75ad6eee661860" exitCode=0 Oct 30 00:54:42.471280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.471226 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" event={"ID":"891703a9-fa46-4691-89ec-7a7777c31f45","Type":"ContainerDied","Data":"02594ba4bb29dce5d94014c888769ab56cc4fff359d51177fd75ad6eee661860"} Oct 30 00:54:42.797808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.796729 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:42.797808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.796790 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:42.918460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.918393 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:54:42.961106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:42.961067 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053247 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053349 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053450 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053499 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053550 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053583 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053643 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053704 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053918 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054006 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054060 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054111 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054164 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054213 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054266 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054303 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\") pod \"891703a9-fa46-4691-89ec-7a7777c31f45\" (UID: \"891703a9-fa46-4691-89ec-7a7777c31f45\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054337 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\") pod \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\" (UID: \"7a9c2842-ed5c-4bf1-a9b2-980dca377beb\") " Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.053919 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.055537 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891703a9-fa46-4691-89ec-7a7777c31f45-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054198 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.055609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.054235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.058544 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.058796 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.058891 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059008 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059146 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059916 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.059966 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.060183 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.063081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.062081 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.063081 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.062796 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:43.063431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.063288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "kube-api-access-8njhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.063620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.063576 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy" pod "891703a9-fa46-4691-89ec-7a7777c31f45" (UID: "891703a9-fa46-4691-89ec-7a7777c31f45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.063620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.063584 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn" pod "7a9c2842-ed5c-4bf1-a9b2-980dca377beb" (UID: "7a9c2842-ed5c-4bf1-a9b2-980dca377beb"). InnerVolumeSpecName "kube-api-access-nflfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.077204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.076895 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:54:43.083537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.083485 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:54:43.158238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158200 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158238 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8njhk\" (UniqueName: \"kubernetes.io/projected/891703a9-fa46-4691-89ec-7a7777c31f45-kube-api-access-8njhk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158253 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158270 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158283 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158297 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158308 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158319 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158332 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflfn\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-kube-api-access-nflfn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158343 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158356 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158368 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158381 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a9c2842-ed5c-4bf1-a9b2-980dca377beb-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158394 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158407 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891703a9-fa46-4691-89ec-7a7777c31f45-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158420 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.158507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.158432 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891703a9-fa46-4691-89ec-7a7777c31f45-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260390 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260533 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260570 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260627 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260659 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260691 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260752 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.260949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.260829 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.261518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261088 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:43.261518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261243 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.261518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261348 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.261518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261358 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.261518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:43.261793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.261532 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262092 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262138 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262188 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262249 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262294 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262328 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262369 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262430 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262470 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\") pod \"50729a04-4c87-429d-8354-4338ac1197e1\" (UID: \"50729a04-4c87-429d-8354-4338ac1197e1\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262555 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\") pod \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\" (UID: \"3d2aeb56-0794-47d6-b61a-ad60e52e47c8\") " Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262616 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.262753 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.263348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263156 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263550 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50729a04-4c87-429d-8354-4338ac1197e1-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263571 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263588 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263603 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263617 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263630 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263644 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263657 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.263670 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.264960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.264670 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.267223 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.267198 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "kube-api-access-p565q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.267330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.267274 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.268983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.268314 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "kube-api-access-d5k7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.273683 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.273656 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:43.274258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.274230 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:43.274342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.274329 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:43.275422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.275400 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token" pod "50729a04-4c87-429d-8354-4338ac1197e1" (UID: "50729a04-4c87-429d-8354-4338ac1197e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.276055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.276031 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token" pod "3d2aeb56-0794-47d6-b61a-ad60e52e47c8" (UID: "3d2aeb56-0794-47d6-b61a-ad60e52e47c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:43.365029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365002 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50729a04-4c87-429d-8354-4338ac1197e1-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365025 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365039 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365051 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365063 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5k7q\" (UniqueName: \"kubernetes.io/projected/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-kube-api-access-d5k7q\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365075 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50729a04-4c87-429d-8354-4338ac1197e1-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365088 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2aeb56-0794-47d6-b61a-ad60e52e47c8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365102 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p565q\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-kube-api-access-p565q\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.365438 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.365117 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50729a04-4c87-429d-8354-4338ac1197e1-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:43.525200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.525132 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" Oct 30 00:54:43.525737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.525701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j" event={"ID":"891703a9-fa46-4691-89ec-7a7777c31f45","Type":"ContainerDied","Data":"1a7ac1fefacd1625652be8adbcda9b58ce8f581899c68c813145bc8fe59a657d"} Oct 30 00:54:43.525832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.525815 5938 scope.go:122] "RemoveContainer" containerID="02594ba4bb29dce5d94014c888769ab56cc4fff359d51177fd75ad6eee661860" Oct 30 00:54:43.528420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.528390 5938 generic.go:356] "Generic (PLEG): container finished" podID="50729a04-4c87-429d-8354-4338ac1197e1" containerID="16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a" exitCode=0 Oct 30 00:54:43.528594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.528570 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" event={"ID":"50729a04-4c87-429d-8354-4338ac1197e1","Type":"ContainerDied","Data":"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a"} Oct 30 00:54:43.528663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.528623 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" event={"ID":"50729a04-4c87-429d-8354-4338ac1197e1","Type":"ContainerDied","Data":"3e24f90079921ba104fd938e109f58e486525839a8e9bdac17e238a051e0409d"} Oct 30 00:54:43.528742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.528726 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd" Oct 30 00:54:43.530931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.530904 5938 generic.go:356] "Generic (PLEG): container finished" podID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerID="0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325" exitCode=0 Oct 30 00:54:43.531064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.531043 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" event={"ID":"7a9c2842-ed5c-4bf1-a9b2-980dca377beb","Type":"ContainerDied","Data":"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325"} Oct 30 00:54:43.531117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.531094 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" event={"ID":"7a9c2842-ed5c-4bf1-a9b2-980dca377beb","Type":"ContainerDied","Data":"9f247cafb3c0a05711d1ab4553eafed2a042aa7f0c77fb1e8f6d9613f9f9559f"} Oct 30 00:54:43.531198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.531183 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n" Oct 30 00:54:43.540948 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.540924 5938 scope.go:122] "RemoveContainer" containerID="16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a" Oct 30 00:54:43.541174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.541151 5938 generic.go:356] "Generic (PLEG): container finished" podID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerID="b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78" exitCode=0 Oct 30 00:54:43.541247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.541193 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" event={"ID":"3d2aeb56-0794-47d6-b61a-ad60e52e47c8","Type":"ContainerDied","Data":"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78"} Oct 30 00:54:43.541247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.541234 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" event={"ID":"3d2aeb56-0794-47d6-b61a-ad60e52e47c8","Type":"ContainerDied","Data":"17ca2219ba54d0c0107ad540d63df50f9c58e471f7b83f66948bd29307583627"} Oct 30 00:54:43.541356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.541321 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd" Oct 30 00:54:43.568724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.568711 5938 scope.go:122] "RemoveContainer" containerID="16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a" Oct 30 00:54:43.569140 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:43.569124 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a\": container with ID starting with 16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a not found: ID does not exist" containerID="16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a" Oct 30 00:54:43.569305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.569288 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a"} err="failed to get container status \"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a\": rpc error: code = NotFound desc = could not find container \"16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a\": container with ID starting with 16e1aa57f92de1e5e36e2e69533a9389c1ce5ed09294b903fe3d2e938abf018a not found: ID does not exist" Oct 30 00:54:43.569386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.569378 5938 scope.go:122] "RemoveContainer" containerID="0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325" Oct 30 00:54:43.580390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.580351 5938 scope.go:122] "RemoveContainer" containerID="0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325" Oct 30 00:54:43.580649 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:43.580628 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325\": container with ID starting with 0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325 not found: ID does not exist" containerID="0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325" Oct 30 00:54:43.580733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.580657 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325"} err="failed to get container status \"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325\": rpc error: code = NotFound desc = could not find container \"0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325\": container with ID starting with 0eba392d0eaaaa6f8693f8416fcdda8146622a073eae4f562b63a3921d8a3325 not found: ID does not exist" Oct 30 00:54:43.580733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.580680 5938 scope.go:122] "RemoveContainer" containerID="b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78" Oct 30 00:54:43.626119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.626101 5938 scope.go:122] "RemoveContainer" containerID="b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78" Oct 30 00:54:43.626460 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:43.626440 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78\": container with ID starting with b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78 not found: ID does not exist" containerID="b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78" Oct 30 00:54:43.626515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.626469 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78"} err="failed to get container status \"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78\": rpc error: code = NotFound desc = could not find container \"b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78\": container with ID starting with b8ae61553b9414567ee847fa8e0210e69a5f8a5a4b102fb9081f92368424aa78 not found: ID does not exist" Oct 30 00:54:43.627190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.627174 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:54:43.631351 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.631319 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-gjq9j"] Oct 30 00:54:43.673499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.673467 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:54:43.679385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.679359 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-m95bd"] Oct 30 00:54:43.713492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.713457 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:54:43.718327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.718304 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-586bd"] Oct 30 00:54:43.750017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.749992 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:54:43.755794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.755765 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-4pk2n"] Oct 30 00:54:43.788089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.788033 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 00:54:43.788089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.788079 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 00:54:43.878799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.878769 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" path="/var/lib/kubelet/pods/3d2aeb56-0794-47d6-b61a-ad60e52e47c8/volumes" Oct 30 00:54:43.879432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.879412 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50729a04-4c87-429d-8354-4338ac1197e1" path="/var/lib/kubelet/pods/50729a04-4c87-429d-8354-4338ac1197e1/volumes" Oct 30 00:54:43.880055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.880036 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" path="/var/lib/kubelet/pods/7a9c2842-ed5c-4bf1-a9b2-980dca377beb/volumes" Oct 30 00:54:43.881102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:43.881084 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" path="/var/lib/kubelet/pods/891703a9-fa46-4691-89ec-7a7777c31f45/volumes" Oct 30 00:54:44.239049 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.239018 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:44.373047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.373010 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:44.691953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.691855 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:44.791704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.791680 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:44.792102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.792082 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:44.792807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.792789 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:54:44.838190 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:44.838165 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:46.308998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.308949 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:46.495991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.495947 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:46.547716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.547685 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 00:54:46.548475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548457 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" Oct 30 00:54:46.548529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548477 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" Oct 30 00:54:46.548529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548490 5938 cpu_manager.go:414] "containerMap: removing container" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" Oct 30 00:54:46.548529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548495 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" Oct 30 00:54:46.548529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548519 5938 cpu_manager.go:414] "containerMap: removing container" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" Oct 30 00:54:46.548529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548523 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548540 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548545 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548657 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="50729a04-4c87-429d-8354-4338ac1197e1" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548667 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2aeb56-0794-47d6-b61a-ad60e52e47c8" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548679 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9c2842-ed5c-4bf1-a9b2-980dca377beb" containerName="istio-proxy" Oct 30 00:54:46.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.548686 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="891703a9-fa46-4691-89ec-7a7777c31f45" containerName="istio-proxy" Oct 30 00:54:46.552586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.552569 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.585907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.585884 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 00:54:46.590345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.590317 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6444d7-dockercfg-h9qjd\"" Oct 30 00:54:46.590449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.590348 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6444d7\"" Oct 30 00:54:46.663887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.649704 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 00:54:46.663887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.660340 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:46.703975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.696742 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:46.738894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.733131 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzr7g\" (UniqueName: \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.738894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.733566 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.738894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.737907 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 00:54:46.859239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.859205 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d5v\" (UniqueName: \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:46.941714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.941679 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr7g\" (UniqueName: \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.942401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.942373 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:46.942931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.942914 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.943991 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.943765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:46.979735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:46.979676 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr7g\" (UniqueName: \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\") pod \"payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:47.045925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.045857 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d5v\" (UniqueName: \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:47.046098 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.046036 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:47.046779 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.046760 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:47.070794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.070742 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d5v\" (UniqueName: \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\") pod \"payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:47.166101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.166070 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:47.212433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.212406 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:47.469673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.469609 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 00:54:47.470064 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:47.469947 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62df9578_ae21_4964_8f31_55a84641a57e.slice/crio-70fd9c479807fb772ad2217428d0978afed7f27d661c91c2dac9c4c2078288a2 WatchSource:0}: Error finding container 70fd9c479807fb772ad2217428d0978afed7f27d661c91c2dac9c4c2078288a2: Status 404 returned error can't find the container with id 70fd9c479807fb772ad2217428d0978afed7f27d661c91c2dac9c4c2078288a2 Oct 30 00:54:47.550270 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:47.550245 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8e13af_9c87_4f40_9bee_3d97de22c836.slice/crio-b5cbe1c2434c84614f439ebacdc6b096c0f7585976d4b153ff191a2a7b4dedd2 WatchSource:0}: Error finding container b5cbe1c2434c84614f439ebacdc6b096c0f7585976d4b153ff191a2a7b4dedd2: Status 404 returned error can't find the container with id b5cbe1c2434c84614f439ebacdc6b096c0f7585976d4b153ff191a2a7b4dedd2 Oct 30 00:54:47.550270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.550256 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 00:54:47.580919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.580892 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" event={"ID":"6e8e13af-9c87-4f40-9bee-3d97de22c836","Type":"ContainerStarted","Data":"b5cbe1c2434c84614f439ebacdc6b096c0f7585976d4b153ff191a2a7b4dedd2"} Oct 30 00:54:47.582821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.582795 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" event={"ID":"62df9578-ae21-4964-8f31-55a84641a57e","Type":"ContainerStarted","Data":"9bec50c7c58ab67f256e97fe30db8d01726c1093b2c03583e5885e1f2300cd09"} Oct 30 00:54:47.582936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.582845 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" event={"ID":"62df9578-ae21-4964-8f31-55a84641a57e","Type":"ContainerStarted","Data":"70fd9c479807fb772ad2217428d0978afed7f27d661c91c2dac9c4c2078288a2"} Oct 30 00:54:47.583013 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.582953 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:47.625680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:47.625630 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" podStartSLOduration=1.625614107 podStartE2EDuration="1.625614107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:47.621454797 +0000 UTC m=+1339.861689912" watchObservedRunningTime="2026-10-30 00:54:47.625614107 +0000 UTC m=+1339.865849221" Oct 30 00:54:48.380251 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:48.380213 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:48.593372 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:48.593336 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" event={"ID":"6e8e13af-9c87-4f40-9bee-3d97de22c836","Type":"ContainerStarted","Data":"4c25506101819f706dbe8750dae98514be11d68ac8f606c0542f4f0744273885"} Oct 30 00:54:48.593805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:48.593496 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:48.636446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:48.636364 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" podStartSLOduration=2.6363498400000003 podStartE2EDuration="2.63634984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:48.633019962 +0000 UTC m=+1340.873255106" watchObservedRunningTime="2026-10-30 00:54:48.63634984 +0000 UTC m=+1340.876584953" Oct 30 00:54:50.057602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:50.057561 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc"] Oct 30 00:54:50.065403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:50.065377 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88289bd-zhrsc"] Oct 30 00:54:50.176915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:50.176880 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:50.377801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:50.377772 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:50.523487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:50.523433 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:51.796194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.796166 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:54:51.796581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.796375 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" containerID="cri-o://87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5" gracePeriod=30 Oct 30 00:54:51.807612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.807589 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:54:51.807816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.807796 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" containerID="cri-o://a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b" gracePeriod=30 Oct 30 00:54:51.811707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.811682 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:54:51.811920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.811891 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" containerID="cri-o://f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532" gracePeriod=30 Oct 30 00:54:51.819058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.819036 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:54:51.819250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.819227 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" containerID="cri-o://0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154" gracePeriod=30 Oct 30 00:54:51.879304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:51.879277 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef439a8-55e3-417b-b344-4122ce0c81b9" path="/var/lib/kubelet/pods/3ef439a8-55e3-417b-b344-4122ce0c81b9/volumes" Oct 30 00:54:52.096325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:52.096219 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:52.479473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:52.479443 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:54:52.479635 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:52.479509 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:54:53.763483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:53.763425 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:55.095907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.093738 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp"] Oct 30 00:54:55.098965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.098943 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.114204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.113920 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:54:55.114204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.113967 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:54:55.115729 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.115705 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp"] Oct 30 00:54:55.119073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.119057 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:54:55.119187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.119156 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:54:55.238834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.238797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.239041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.238929 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.239141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.239070 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmf4\" (UniqueName: \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.341470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.341432 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.341634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.341522 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.341634 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.341628 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmf4\" (UniqueName: \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.342153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.342122 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.344010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.343985 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.364004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.363977 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmf4\" (UniqueName: \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\") pod \"maas-api-revoke-sub-c72ea9b2-5snbp\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.418348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.418310 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:55.539544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.538177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:55.676765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.676681 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:55.794141 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:55.794112 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp"] Oct 30 00:54:55.796435 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:54:55.796396 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832a28fd_9a64_43d7_8d6a_97b4ad5c2946.slice/crio-49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc WatchSource:0}: Error finding container 49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc: Status 404 returned error can't find the container with id 49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc Oct 30 00:54:56.139514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:56.139482 5938 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:54:56.140029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:56.139530 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:54:56.675824 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:56.675779 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" event={"ID":"832a28fd-9a64-43d7-8d6a-97b4ad5c2946","Type":"ContainerStarted","Data":"db22a9bc52e0fab47c40607fe2738ac36560c0ffc3283d90a8d59bf5d8477c7e"} Oct 30 00:54:56.676022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:56.675887 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" event={"ID":"832a28fd-9a64-43d7-8d6a-97b4ad5c2946","Type":"ContainerStarted","Data":"49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc"} Oct 30 00:54:56.718734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:56.718690 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" podStartSLOduration=1.718677059 podStartE2EDuration="1.718677059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:56.713741893 +0000 UTC m=+1348.953977008" watchObservedRunningTime="2026-10-30 00:54:56.718677059 +0000 UTC m=+1348.958912172" Oct 30 00:54:57.168736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.168666 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.321840 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.321999 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322065 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322171 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322235 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322290 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322436 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.323892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.322517 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\") pod \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\" (UID: \"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c\") " Oct 30 00:54:57.325411 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.325191 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:57.325940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.325638 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.326568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.326534 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.326765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.326745 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.328749 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.328717 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.333412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.333375 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.333942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.333744 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:54:57.334461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.334437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.335368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.335345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:57.337581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.337540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt" pod "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" (UID: "289013c3-a8f4-4dac-bcf9-1d7b293f2e2c"). InnerVolumeSpecName "kube-api-access-nzzzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.346791 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.346771 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:54:57.376067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.376047 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:54:57.376650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.376619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430410 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430448 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430465 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430479 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430495 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430508 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430522 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzzzt\" (UniqueName: \"kubernetes.io/projected/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-kube-api-access-nzzzt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430538 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.430551 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.532280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532246 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532298 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532338 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532359 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532404 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532436 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532477 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532535 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.532585 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532558 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.532704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532633 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532665 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532705 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532726 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532811 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.532851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532841 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.532982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532896 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.532982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.532982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532951 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.533073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.532982 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\") pod \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\" (UID: \"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3\") " Oct 30 00:54:57.533073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533024 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.533073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533045 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.533073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.533195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.533195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533134 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\") pod \"7b559555-d33f-4c3e-9756-d671ec7837ce\" (UID: \"7b559555-d33f-4c3e-9756-d671ec7837ce\") " Oct 30 00:54:57.533195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533164 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\") pod \"3ae85fb7-5635-4154-a571-6f1c40a533d2\" (UID: \"3ae85fb7-5635-4154-a571-6f1c40a533d2\") " Oct 30 00:54:57.533758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.533688 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:57.534293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.534256 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:57.534529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.534444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.534647 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.534490 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.535402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.534525 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.535554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.534760 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.535654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.535270 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.536782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.536188 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.536902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.536794 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.538257 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.538359 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "kube-api-access-rhrxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.538498 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.538681 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539254 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539384 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539408 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539423 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539438 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539472 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539485 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539498 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539513 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b559555-d33f-4c3e-9756-d671ec7837ce-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539547 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539563 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrxz\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-kube-api-access-rhrxz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539577 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.539592 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.540089 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.540409 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.540662 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.543147 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "kube-api-access-h96r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.543241 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "kube-api-access-bfjhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.543339 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.543601 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.543644 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:57.544177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.544064 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:57.545542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.544538 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:54:57.546499 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.546476 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token" pod "3ae85fb7-5635-4154-a571-6f1c40a533d2" (UID: "3ae85fb7-5635-4154-a571-6f1c40a533d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.547069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.547031 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:54:57.547985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.547948 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token" pod "7b559555-d33f-4c3e-9756-d671ec7837ce" (UID: "7b559555-d33f-4c3e-9756-d671ec7837ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.548998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.548975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token" pod "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" (UID: "5dba1ce4-fc03-4f62-bc06-ead6a364e3c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:57.640874 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640829 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.640874 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640858 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96r6\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-kube-api-access-h96r6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.640874 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640882 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640893 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfjhl\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-kube-api-access-bfjhl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640906 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640914 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640923 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640933 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640941 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640949 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640956 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae85fb7-5635-4154-a571-6f1c40a533d2-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640963 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b559555-d33f-4c3e-9756-d671ec7837ce-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640971 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae85fb7-5635-4154-a571-6f1c40a533d2-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.641094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.640979 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:57.711162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.711129 5938 generic.go:356] "Generic (PLEG): container finished" podID="832a28fd-9a64-43d7-8d6a-97b4ad5c2946" containerID="db22a9bc52e0fab47c40607fe2738ac36560c0ffc3283d90a8d59bf5d8477c7e" exitCode=0 Oct 30 00:54:57.711370 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.711210 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" event={"ID":"832a28fd-9a64-43d7-8d6a-97b4ad5c2946","Type":"ContainerDied","Data":"db22a9bc52e0fab47c40607fe2738ac36560c0ffc3283d90a8d59bf5d8477c7e"} Oct 30 00:54:57.712821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.712796 5938 generic.go:356] "Generic (PLEG): container finished" podID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerID="a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b" exitCode=0 Oct 30 00:54:57.712953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.712839 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" Oct 30 00:54:57.712953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.712893 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" event={"ID":"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c","Type":"ContainerDied","Data":"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b"} Oct 30 00:54:57.712953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.712949 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq" event={"ID":"289013c3-a8f4-4dac-bcf9-1d7b293f2e2c","Type":"ContainerDied","Data":"0fdcffb140c60347894e3d368fe192d14b98112a98f66df61f4425b67b7c5911"} Oct 30 00:54:57.713086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.712978 5938 scope.go:122] "RemoveContainer" containerID="a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b" Oct 30 00:54:57.715203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.715179 5938 generic.go:356] "Generic (PLEG): container finished" podID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerID="f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532" exitCode=0 Oct 30 00:54:57.715311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.715238 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" Oct 30 00:54:57.715311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.715294 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerDied","Data":"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532"} Oct 30 00:54:57.715439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.715338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b" event={"ID":"7b559555-d33f-4c3e-9756-d671ec7837ce","Type":"ContainerDied","Data":"ab5fc90e4f1edbd64063ed7a4e4a75328796df8a1a0f86ade63e1d01f721895d"} Oct 30 00:54:57.717169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.717130 5938 generic.go:356] "Generic (PLEG): container finished" podID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerID="87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5" exitCode=0 Oct 30 00:54:57.717269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.717196 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" Oct 30 00:54:57.717269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.717208 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" event={"ID":"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3","Type":"ContainerDied","Data":"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5"} Oct 30 00:54:57.717269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.717264 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r" event={"ID":"5dba1ce4-fc03-4f62-bc06-ead6a364e3c3","Type":"ContainerDied","Data":"60e81eaa73c28ec66e2d089476a38bbaeb5c7686efe09475a8ba4b924c702a8b"} Oct 30 00:54:57.719737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.719716 5938 generic.go:356] "Generic (PLEG): container finished" podID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerID="0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154" exitCode=0 Oct 30 00:54:57.719834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.719740 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" event={"ID":"3ae85fb7-5635-4154-a571-6f1c40a533d2","Type":"ContainerDied","Data":"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154"} Oct 30 00:54:57.719834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.719775 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" Oct 30 00:54:57.719834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.719802 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m" event={"ID":"3ae85fb7-5635-4154-a571-6f1c40a533d2","Type":"ContainerDied","Data":"a677d4e04842e7ea6572a1eb768aca14b6e3eb69c728c33eac6cea84dc5df5d4"} Oct 30 00:54:57.728289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.728273 5938 scope.go:122] "RemoveContainer" containerID="a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b" Oct 30 00:54:57.728565 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:57.728548 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b\": container with ID starting with a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b not found: ID does not exist" containerID="a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b" Oct 30 00:54:57.728642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.728571 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b"} err="failed to get container status \"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b\": rpc error: code = NotFound desc = could not find container \"a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b\": container with ID starting with a179c3592055941984d81a20c5768522d35769be2c96dfd247df50b5bdd63d5b not found: ID does not exist" Oct 30 00:54:57.728642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.728588 5938 scope.go:122] "RemoveContainer" containerID="f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532" Oct 30 00:54:57.739771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.739755 5938 scope.go:122] "RemoveContainer" containerID="29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a" Oct 30 00:54:57.753576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.753561 5938 scope.go:122] "RemoveContainer" containerID="f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532" Oct 30 00:54:57.753875 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:57.753842 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532\": container with ID starting with f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532 not found: ID does not exist" containerID="f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532" Oct 30 00:54:57.753932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.753887 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532"} err="failed to get container status \"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532\": rpc error: code = NotFound desc = could not find container \"f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532\": container with ID starting with f81596998e921444d97032f4b24860d55751b26be51c30498751a4dded417532 not found: ID does not exist" Oct 30 00:54:57.753932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.753911 5938 scope.go:122] "RemoveContainer" containerID="29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a" Oct 30 00:54:57.754185 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:57.754165 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a\": container with ID starting with 29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a not found: ID does not exist" containerID="29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a" Oct 30 00:54:57.754240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.754190 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a"} err="failed to get container status \"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a\": rpc error: code = NotFound desc = could not find container \"29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a\": container with ID starting with 29a1bb62195eb2201173365a9310eea334b1c27aeab1cb53eec0e0f359a6698a not found: ID does not exist" Oct 30 00:54:57.754240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.754205 5938 scope.go:122] "RemoveContainer" containerID="87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5" Oct 30 00:54:57.764034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.764004 5938 scope.go:122] "RemoveContainer" containerID="87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5" Oct 30 00:54:57.764282 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:57.764266 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5\": container with ID starting with 87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5 not found: ID does not exist" containerID="87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5" Oct 30 00:54:57.764333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.764290 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5"} err="failed to get container status \"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5\": rpc error: code = NotFound desc = could not find container \"87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5\": container with ID starting with 87399f59e6ab27d07edb791c9b31361c0727af6c4aeaf3010e7f1cd70df461d5 not found: ID does not exist" Oct 30 00:54:57.764333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.764304 5938 scope.go:122] "RemoveContainer" containerID="0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154" Oct 30 00:54:57.773618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.773601 5938 scope.go:122] "RemoveContainer" containerID="0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154" Oct 30 00:54:57.773918 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:57.773901 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154\": container with ID starting with 0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154 not found: ID does not exist" containerID="0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154" Oct 30 00:54:57.773985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.773925 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154"} err="failed to get container status \"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154\": rpc error: code = NotFound desc = could not find container \"0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154\": container with ID starting with 0f75f0e100deca963d30604b9027e3ad7c14c4f773b44eb0164f396f91b1e154 not found: ID does not exist" Oct 30 00:54:57.816613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.816530 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:54:57.820526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.820500 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-r98qq"] Oct 30 00:54:57.846245 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.846190 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:54:57.849818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.849797 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5c75b"] Oct 30 00:54:57.878399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.878079 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" path="/var/lib/kubelet/pods/289013c3-a8f4-4dac-bcf9-1d7b293f2e2c/volumes" Oct 30 00:54:57.878816 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.878792 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" path="/var/lib/kubelet/pods/7b559555-d33f-4c3e-9756-d671ec7837ce/volumes" Oct 30 00:54:57.882618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.882365 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:54:57.909018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.908977 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-sb97m"] Oct 30 00:54:57.928073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.928046 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:54:57.934107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:57.934081 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-5mg6r"] Oct 30 00:54:58.611494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:58.611469 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 00:54:58.845105 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:54:58.845064 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:54:58.969126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:58.969102 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:59.032889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.032328 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:59.062564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.062516 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\") pod \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " Oct 30 00:54:59.062744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.062605 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\" (UniqueName: \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\") pod \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " Oct 30 00:54:59.062744 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.062693 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\") pod \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\" (UID: \"832a28fd-9a64-43d7-8d6a-97b4ad5c2946\") " Oct 30 00:54:59.064628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.064596 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca" pod "832a28fd-9a64-43d7-8d6a-97b4ad5c2946" (UID: "832a28fd-9a64-43d7-8d6a-97b4ad5c2946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:54:59.070560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.069915 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca" pod "832a28fd-9a64-43d7-8d6a-97b4ad5c2946" (UID: "832a28fd-9a64-43d7-8d6a-97b4ad5c2946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:59.075492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.075454 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4" pod "832a28fd-9a64-43d7-8d6a-97b4ad5c2946" (UID: "832a28fd-9a64-43d7-8d6a-97b4ad5c2946"). InnerVolumeSpecName "kube-api-access-6cmf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:59.165693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.165666 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:59.165693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.165691 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmf4\" (UniqueName: \"kubernetes.io/projected/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-kube-api-access-6cmf4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:59.165928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.165705 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832a28fd-9a64-43d7-8d6a-97b4ad5c2946-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:54:59.611446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.611418 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 00:54:59.744209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.744166 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" Oct 30 00:54:59.744209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.744200 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp" event={"ID":"832a28fd-9a64-43d7-8d6a-97b4ad5c2946","Type":"ContainerDied","Data":"49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc"} Oct 30 00:54:59.744706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.744248 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49966f8fd3d00c5a9524fa908600f07e995058cbcd1aec8529d7f62e796aecfc" Oct 30 00:54:59.878982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.878897 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" path="/var/lib/kubelet/pods/3ae85fb7-5635-4154-a571-6f1c40a533d2/volumes" Oct 30 00:54:59.879464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:54:59.879450 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" path="/var/lib/kubelet/pods/5dba1ce4-fc03-4f62-bc06-ead6a364e3c3/volumes" Oct 30 00:55:00.809439 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:00.809397 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:01.503886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:01.502479 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:02.510693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:02.510660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:02.653738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:02.653702 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:02.819942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:02.819234 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:03.224082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:03.224049 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:04.648073 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:04.648022 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:05.573314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:05.573282 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:06.246975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.246941 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:06.805714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.805680 5938 generic.go:356] "Generic (PLEG): container finished" podID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerID="354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f" exitCode=0 Oct 30 00:55:06.805927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.805746 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerDied","Data":"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f"} Oct 30 00:55:06.806285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.806267 5938 scope.go:122] "RemoveContainer" containerID="354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f" Oct 30 00:55:06.808099 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.808073 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:06.808183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.808140 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:06.808183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:06.808174 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:07.744182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:07.744135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:07.825146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:07.825113 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerStarted","Data":"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc"} Oct 30 00:55:08.610592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:08.610555 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:55:08.610781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:08.610707 5938 patch_prober.go:29] interesting pod/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 00:55:08.610781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:08.610764 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 00:55:09.371942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:09.371909 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:09.610466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:09.610429 5938 patch_prober.go:29] interesting pod/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 00:55:09.610637 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:09.610511 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 00:55:09.704723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:09.704604 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:10.611382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:10.611347 5938 patch_prober.go:29] interesting pod/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 00:55:10.611800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:10.611404 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 00:55:11.134168 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:11.134126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:11.611435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:11.611402 5938 patch_prober.go:29] interesting pod/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 00:55:11.611830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:11.611467 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 00:55:12.266734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.266692 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.507886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.507732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:12.611382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.611354 5938 patch_prober.go:29] interesting pod/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 00:55:12.611545 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.611405 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924107 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924804 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924817 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924830 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924835 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924852 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924857 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924885 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924893 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924952 5938 cpu_manager.go:414] "containerMap: removing container" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924961 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924983 5938 cpu_manager.go:414] "containerMap: removing container" podUID="832a28fd-9a64-43d7-8d6a-97b4ad5c2946" containerName="revoke-keys" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.924990 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832a28fd-9a64-43d7-8d6a-97b4ad5c2946" containerName="revoke-keys" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925206 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="289013c3-a8f4-4dac-bcf9-1d7b293f2e2c" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925223 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="832a28fd-9a64-43d7-8d6a-97b4ad5c2946" containerName="revoke-keys" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925241 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925254 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae85fb7-5635-4154-a571-6f1c40a533d2" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925268 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b559555-d33f-4c3e-9756-d671ec7837ce" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.925278 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dba1ce4-fc03-4f62-bc06-ead6a364e3c3" containerName="istio-proxy" Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.929302 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:55:12.932938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.930954 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:12.947089 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.947024 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fa8dd0-openshift-default-dockercfg-hwm8h\"" Oct 30 00:55:12.975096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:12.975026 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:13.053588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053561 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.053746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053613 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.053746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpjk\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.053822 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053785 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.053857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.053993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.053979 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.054038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.054024 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.054124 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.054111 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.054202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.054189 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.131886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.130939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:13.156002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.155961 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.156308 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.156286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.156551 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.156533 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.156708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.156693 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.156835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.156820 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.157084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.157063 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.157230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.157214 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.157392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.157376 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.157546 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.157528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpjk\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.166885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.158477 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.168227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.168201 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.170055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.168586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.170223 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.168809 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.170361 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.169049 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.184509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.184440 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpjk\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.184927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.184906 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.241888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.230570 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.243019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.242619 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\") pod \"e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.251622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.251375 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:13.642628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.642596 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:55:13.683904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.683836 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:55:13.732528 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.732503 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:55:13.733438 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:13.733415 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9308ee3_b32d_4860_ba78_9c943a245daf.slice/crio-bc6a243b10fd4ac8095a4858cf159404b512826d7dc77cbda3ae3510d1a2f31b WatchSource:0}: Error finding container bc6a243b10fd4ac8095a4858cf159404b512826d7dc77cbda3ae3510d1a2f31b: Status 404 returned error can't find the container with id bc6a243b10fd4ac8095a4858cf159404b512826d7dc77cbda3ae3510d1a2f31b Oct 30 00:55:13.736102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.736066 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:13.736159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.736128 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:13.736194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.736159 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:13.951282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.951240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerStarted","Data":"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10"} Oct 30 00:55:13.951483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.951337 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerStarted","Data":"bc6a243b10fd4ac8095a4858cf159404b512826d7dc77cbda3ae3510d1a2f31b"} Oct 30 00:55:13.953139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.953111 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:55:13.996422 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:13.996313 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podStartSLOduration=1.996295344 podStartE2EDuration="1.996295344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:13.994044649 +0000 UTC m=+1366.234279768" watchObservedRunningTime="2026-10-30 00:55:13.996295344 +0000 UTC m=+1366.236530460" Oct 30 00:55:14.255937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.252791 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:14.265155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.262963 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:14.721544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.721513 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 00:55:14.725147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.725130 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.745310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.745288 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:55:14.745472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.745393 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:55:14.747182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.747158 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 00:55:14.754362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.754343 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6444d7-serving-cert\"" Oct 30 00:55:14.754448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.754422 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:55:14.755649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.755632 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x95xl\"" Oct 30 00:55:14.822706 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.822678 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rbl\" (UniqueName: \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.822908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.822886 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.823009 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.822993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.823088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.823077 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.848070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.848044 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.924584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.924560 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rbl\" (UniqueName: \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.924747 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.924728 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.924853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.924826 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.925002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.924983 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.925489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.925465 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.927155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.927128 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.927238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.927214 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:14.988029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:14.985607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rbl\" (UniqueName: \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\") pod \"maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:15.010924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.008954 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:15.010924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.010726 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:15.010924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.010809 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:55:15.040025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.039995 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:15.366658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.366631 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 00:55:15.367566 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:15.367534 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7629ae34_a780_47a9_a2b5_753d54ab3fa6.slice/crio-61b80c51e8fd75765bb9a3f8bed713f6a013ad88c40de67700f135b6e461d416 WatchSource:0}: Error finding container 61b80c51e8fd75765bb9a3f8bed713f6a013ad88c40de67700f135b6e461d416: Status 404 returned error can't find the container with id 61b80c51e8fd75765bb9a3f8bed713f6a013ad88c40de67700f135b6e461d416 Oct 30 00:55:15.917315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:15.917281 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:16.062476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:16.062437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" event={"ID":"7629ae34-a780-47a9-a2b5-753d54ab3fa6","Type":"ContainerStarted","Data":"61b80c51e8fd75765bb9a3f8bed713f6a013ad88c40de67700f135b6e461d416"} Oct 30 00:55:16.528255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:16.528216 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:16.675135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:16.675096 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:17.394117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:17.394084 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:17.539050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:17.538962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:17.689103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:17.689023 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:18.120282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.120227 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:18.612162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.612113 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:18.757911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.757879 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:55:18.761568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.761512 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.764173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.764139 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:18.776118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.776097 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fa8dd0\"" Oct 30 00:55:18.788498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.788457 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fa8dd0-dockercfg-85n56\"" Oct 30 00:55:18.792768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.792745 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:55:18.823730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.823695 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvms\" (UniqueName: \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.824084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.824060 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.952108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.950567 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvms\" (UniqueName: \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.952108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.950681 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.952108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.951668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:18.991916 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:18.984635 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvms\" (UniqueName: \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\") pod \"payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:19.012509 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.010681 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:55:19.036886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.036024 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:55:19.036886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.036148 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.067097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.066751 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7vs\" (UniqueName: \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.067187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.067135 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.077618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.077216 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:19.133078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.131942 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" event={"ID":"7629ae34-a780-47a9-a2b5-753d54ab3fa6","Type":"ContainerStarted","Data":"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51"} Oct 30 00:55:19.133377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.133357 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:19.170513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.169527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.170513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.169710 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7vs\" (UniqueName: \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.172758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.172590 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.177624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.177587 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" podStartSLOduration=2.03699013 podStartE2EDuration="5.177574433s" totalImagesPullingTime="3.140584303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:19.177192959 +0000 UTC m=+1371.417428074" watchObservedRunningTime="2026-10-30 00:55:19.177574433 +0000 UTC m=+1371.417809540" Oct 30 00:55:19.210627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.210599 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7vs\" (UniqueName: \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\") pod \"payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.380028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.379995 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:19.421384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.421358 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:55:19.423220 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:19.423189 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3e73c7_dc8f_4c85_8a7c_f8f907a63d6b.slice/crio-2f5255c3b446db55e6bebaa1391d630bb705e87f1199b6ba89e2405ae8153814 WatchSource:0}: Error finding container 2f5255c3b446db55e6bebaa1391d630bb705e87f1199b6ba89e2405ae8153814: Status 404 returned error can't find the container with id 2f5255c3b446db55e6bebaa1391d630bb705e87f1199b6ba89e2405ae8153814 Oct 30 00:55:19.625295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.625260 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:19.702328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:19.702296 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:55:20.142693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.142661 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" event={"ID":"20feefc3-1f0b-456f-a672-b98a21d065f7","Type":"ContainerStarted","Data":"460c3202dd3d1c025a4be3f35f1c578aba64110b77c42af5e607bf94079fe444"} Oct 30 00:55:20.142897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.142723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" event={"ID":"20feefc3-1f0b-456f-a672-b98a21d065f7","Type":"ContainerStarted","Data":"df4c4f7eafcf10ced17f0563c9516027c8a70968c135fdfa85d08c9066ad2198"} Oct 30 00:55:20.142897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.142755 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:20.144722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.144695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" event={"ID":"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b","Type":"ContainerStarted","Data":"0ea2706e7e321484052cbaa1807680f5abf5684be1d8727e6d7ba0815025c0c1"} Oct 30 00:55:20.144845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.144749 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:20.144845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.144773 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" event={"ID":"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b","Type":"ContainerStarted","Data":"2f5255c3b446db55e6bebaa1391d630bb705e87f1199b6ba89e2405ae8153814"} Oct 30 00:55:20.226034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.225987 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" podStartSLOduration=2.225973315 podStartE2EDuration="2.225973315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:20.184116833 +0000 UTC m=+1372.424352019" watchObservedRunningTime="2026-10-30 00:55:20.225973315 +0000 UTC m=+1372.466208428" Oct 30 00:55:20.226260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:20.226241 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" podStartSLOduration=2.226233957 podStartE2EDuration="2.226233957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:20.222321231 +0000 UTC m=+1372.462556343" watchObservedRunningTime="2026-10-30 00:55:20.226233957 +0000 UTC m=+1372.466469069" Oct 30 00:55:21.230097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:21.230065 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:22.073179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.073150 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:55:22.073397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.073355 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" containerID="cri-o://3c5f41319580f0e755d8dd2e86cfc6c5d4cc6fda2ea9b37d1a9aa5dc0aa0626f" gracePeriod=30 Oct 30 00:55:22.085070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.085044 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:55:22.085264 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.085244 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" containerID="cri-o://97c206a165436ec46b8953ff403f35baa0071b422b17a95a2782a283574e06d9" gracePeriod=30 Oct 30 00:55:22.089507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.089473 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:55:22.089724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.089695 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" containerID="cri-o://ec254a66005b5205c80073a5a48af851e19c2067e551e35aed43a7674ca162c4" gracePeriod=30 Oct 30 00:55:22.096243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.096219 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:55:22.096430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.096390 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" containerID="cri-o://25879a21ed10f6c99fbb02b655325f44f3dc0f6556ba957b02b0577fa4ad124f" gracePeriod=30 Oct 30 00:55:22.362031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.362003 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:55:22.362493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.362263 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" containerID="cri-o://c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a" gracePeriod=30 Oct 30 00:55:22.366443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.366416 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:55:22.366624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.366604 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" containerID="cri-o://8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3" gracePeriod=30 Oct 30 00:55:22.372230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.372204 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:55:22.372440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.372417 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" containerID="cri-o://2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11" gracePeriod=30 Oct 30 00:55:22.851617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:22.851489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:23.021302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:23.021267 5938 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:55:23.021468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:23.021341 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:55:24.507899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.507840 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:24.649418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.649386 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:24.794106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.794006 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:24.806143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.806102 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:55:24.806383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.806355 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" containerID="cri-o://27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9" gracePeriod=30 Oct 30 00:55:24.959888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:24.947401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:25.150801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:25.150775 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 00:55:26.658433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:26.658408 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:27.216076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.216046 5938 generic.go:356] "Generic (PLEG): container finished" podID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerID="97c206a165436ec46b8953ff403f35baa0071b422b17a95a2782a283574e06d9" exitCode=0 Oct 30 00:55:27.216238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.216125 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerDied","Data":"97c206a165436ec46b8953ff403f35baa0071b422b17a95a2782a283574e06d9"} Oct 30 00:55:27.216238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.216200 5938 scope.go:122] "RemoveContainer" containerID="6a895a218bd28f0969b39923e0844de7bb22d8f2d319a1c85eda3e288124b62a" Oct 30 00:55:27.222466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.222440 5938 generic.go:356] "Generic (PLEG): container finished" podID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerID="3c5f41319580f0e755d8dd2e86cfc6c5d4cc6fda2ea9b37d1a9aa5dc0aa0626f" exitCode=0 Oct 30 00:55:27.222574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.222547 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" event={"ID":"e206ffd7-069f-4e1a-a4d0-31c5659f436c","Type":"ContainerDied","Data":"3c5f41319580f0e755d8dd2e86cfc6c5d4cc6fda2ea9b37d1a9aa5dc0aa0626f"} Oct 30 00:55:27.224721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.224698 5938 generic.go:356] "Generic (PLEG): container finished" podID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerID="ec254a66005b5205c80073a5a48af851e19c2067e551e35aed43a7674ca162c4" exitCode=0 Oct 30 00:55:27.224975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.224949 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" event={"ID":"213e2306-e63e-478a-9d7d-c1c5038343e3","Type":"ContainerDied","Data":"ec254a66005b5205c80073a5a48af851e19c2067e551e35aed43a7674ca162c4"} Oct 30 00:55:27.227412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.227025 5938 generic.go:356] "Generic (PLEG): container finished" podID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerID="25879a21ed10f6c99fbb02b655325f44f3dc0f6556ba957b02b0577fa4ad124f" exitCode=0 Oct 30 00:55:27.227412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.227088 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" event={"ID":"a34feee0-f121-4b7a-964a-aea2f35c1475","Type":"ContainerDied","Data":"25879a21ed10f6c99fbb02b655325f44f3dc0f6556ba957b02b0577fa4ad124f"} Oct 30 00:55:27.614841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.614812 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:55:27.621300 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.621274 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:55:27.627461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.627433 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:55:27.633227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.633207 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:55:27.701397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.701373 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:55:27.727850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.727823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.728122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.728098 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.728294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.728277 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.728472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.728445 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.729283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729262 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.729394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729327 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.729394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729372 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729451 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729500 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729599 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.729738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729634 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.729738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729688 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.729955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729779 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.729955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729810 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.729955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729875 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.729964 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.730149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730015 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730056 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.730149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730125 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730164 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730230 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730356 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730302 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.730562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730354 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.730562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730416 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730463 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.730562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730562 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730551 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730628 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\") pod \"213e2306-e63e-478a-9d7d-c1c5038343e3\" (UID: \"213e2306-e63e-478a-9d7d-c1c5038343e3\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730704 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730754 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\") pod \"a34feee0-f121-4b7a-964a-aea2f35c1475\" (UID: \"a34feee0-f121-4b7a-964a-aea2f35c1475\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\") pod \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\" (UID: \"e206ffd7-069f-4e1a-a4d0-31c5659f436c\") " Oct 30 00:55:27.730884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.730843 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\") pod \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\" (UID: \"1b06c81b-19fa-4af1-9da1-b9a6dee53e53\") " Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.728146 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.728725 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.731600 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.731852 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.732555 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.732858 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.732980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.733169 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.733182 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.733346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.734143 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.734298 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.740970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.741673 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "kube-api-access-8lzds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.742591 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.743368 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.744326 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.744582 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.744833 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.745034 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.745240 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.745368 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.746089 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.746210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.746156 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.749359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.748246 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.749359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.748494 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.749359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.749109 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.749359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.749173 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "kube-api-access-j6h9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.754169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.754140 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.754281 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.754213 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.757074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.756900 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "kube-api-access-p7nbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.757074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.756970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.757074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.756975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy" pod "1b06c81b-19fa-4af1-9da1-b9a6dee53e53" (UID: "1b06c81b-19fa-4af1-9da1-b9a6dee53e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.757074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.757009 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc" pod "a34feee0-f121-4b7a-964a-aea2f35c1475" (UID: "a34feee0-f121-4b7a-964a-aea2f35c1475"). InnerVolumeSpecName "kube-api-access-g4sjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.757504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.757093 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token" pod "213e2306-e63e-478a-9d7d-c1c5038343e3" (UID: "213e2306-e63e-478a-9d7d-c1c5038343e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.759227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.759202 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo" pod "e206ffd7-069f-4e1a-a4d0-31c5659f436c" (UID: "e206ffd7-069f-4e1a-a4d0-31c5659f436c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.807602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.807579 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:55:27.812045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.812021 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:55:27.833563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.833533 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.833698 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.833626 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.833768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.833703 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.833912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.833877 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.833990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.833961 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.834059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834046 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.834121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834086 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.834179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834124 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.834179 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834154 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\") pod \"6a0b50c8-8bad-4637-89f0-a899b514d204\" (UID: \"6a0b50c8-8bad-4637-89f0-a899b514d204\") " Oct 30 00:55:27.834381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834348 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.834454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.834454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834426 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.834623 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834600 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.834945 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.834919 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835126 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835154 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835169 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nbg\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-kube-api-access-p7nbg\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835184 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835197 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835211 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835225 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835237 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835248 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835260 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835273 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835285 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.835343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.835297 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34feee0-f121-4b7a-964a-aea2f35c1475-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836097 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836124 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836143 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836157 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836170 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836184 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836196 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzds\" (UniqueName: \"kubernetes.io/projected/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-kube-api-access-8lzds\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836209 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836221 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836232 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836243 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836256 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836267 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a0b50c8-8bad-4637-89f0-a899b514d204-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836279 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e206ffd7-069f-4e1a-a4d0-31c5659f436c-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836292 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6h9b\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-kube-api-access-j6h9b\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836304 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836316 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836327 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836340 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4sjc\" (UniqueName: \"kubernetes.io/projected/a34feee0-f121-4b7a-964a-aea2f35c1475-kube-api-access-g4sjc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836353 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836364 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e206ffd7-069f-4e1a-a4d0-31c5659f436c-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836375 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836398 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836409 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836421 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213e2306-e63e-478a-9d7d-c1c5038343e3-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836432 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213e2306-e63e-478a-9d7d-c1c5038343e3-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836443 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34feee0-f121-4b7a-964a-aea2f35c1475-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.836586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836456 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b06c81b-19fa-4af1-9da1-b9a6dee53e53-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.837205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836647 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "kube-api-access-g7577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.837205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.836836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.838075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.838050 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.838341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.838319 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo" pod "6a0b50c8-8bad-4637-89f0-a899b514d204" (UID: "6a0b50c8-8bad-4637-89f0-a899b514d204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.937792 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.937757 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.937832 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.937899 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.937943 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938064 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.937989 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938076 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938267 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938314 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938365 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938414 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938458 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938517 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938597 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\") pod \"cb14d4d8-31e4-4a02-925c-03fc3d350026\" (UID: \"cb14d4d8-31e4-4a02-925c-03fc3d350026\") " Oct 30 00:55:27.938987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.938764 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\") pod \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\" (UID: \"944cdc20-56d4-41bf-9b47-4ee5e82b5d50\") " Oct 30 00:55:27.939569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.939544 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.939721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.939707 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.939840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.939825 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.939952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.939939 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7577\" (UniqueName: \"kubernetes.io/projected/6a0b50c8-8bad-4637-89f0-a899b514d204-kube-api-access-g7577\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:27.940418 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.940394 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.941334 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.941594 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.941767 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.941912 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.941947 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.942343 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.942526 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.943005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.942719 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.944151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.943851 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:27.944904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.944338 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.944904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.944642 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.945062 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.945034 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:27.945254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.945221 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.945714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.945686 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.948906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.947963 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "kube-api-access-h28bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:27.948906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.948061 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy" pod "cb14d4d8-31e4-4a02-925c-03fc3d350026" (UID: "cb14d4d8-31e4-4a02-925c-03fc3d350026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:27.948906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:27.948116 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x" pod "944cdc20-56d4-41bf-9b47-4ee5e82b5d50" (UID: "944cdc20-56d4-41bf-9b47-4ee5e82b5d50"). InnerVolumeSpecName "kube-api-access-qt66x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:28.041295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041265 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041296 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041310 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041324 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041337 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041350 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041364 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041390 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt66x\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-kube-api-access-qt66x\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041413 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041428 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28bl\" (UniqueName: \"kubernetes.io/projected/cb14d4d8-31e4-4a02-925c-03fc3d350026-kube-api-access-h28bl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041441 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb14d4d8-31e4-4a02-925c-03fc3d350026-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041454 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041466 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041479 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041492 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041505 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb14d4d8-31e4-4a02-925c-03fc3d350026-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041519 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb14d4d8-31e4-4a02-925c-03fc3d350026-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.041834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.041532 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944cdc20-56d4-41bf-9b47-4ee5e82b5d50-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:28.092176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.092135 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:55:28.125450 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.125418 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:55:28.241209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.241126 5938 generic.go:356] "Generic (PLEG): container finished" podID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerID="8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3" exitCode=0 Oct 30 00:55:28.241457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.241195 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" Oct 30 00:55:28.241825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.241231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" event={"ID":"6a0b50c8-8bad-4637-89f0-a899b514d204","Type":"ContainerDied","Data":"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3"} Oct 30 00:55:28.241825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.241654 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5" event={"ID":"6a0b50c8-8bad-4637-89f0-a899b514d204","Type":"ContainerDied","Data":"9c1ebfb36b727915c93f4b13557c2691b42c67a3912f3e044794823b75e29b5d"} Oct 30 00:55:28.241825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.241695 5938 scope.go:122] "RemoveContainer" containerID="8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3" Oct 30 00:55:28.253946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.253920 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" event={"ID":"e206ffd7-069f-4e1a-a4d0-31c5659f436c","Type":"ContainerDied","Data":"9b6b555ee7e2f41648a1af772f4fffab43794f1a26e4dc428c86c47ff47ca02e"} Oct 30 00:55:28.254199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.254182 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf" Oct 30 00:55:28.257514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.257485 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" event={"ID":"213e2306-e63e-478a-9d7d-c1c5038343e3","Type":"ContainerDied","Data":"3f0e45795fae9b05bc319888f8049a276d84a2011d75c0b96c11bdbafc43913e"} Oct 30 00:55:28.257631 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.257614 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb" Oct 30 00:55:28.259136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.259119 5938 scope.go:122] "RemoveContainer" containerID="8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3" Oct 30 00:55:28.259583 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:28.259554 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3\": container with ID starting with 8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3 not found: ID does not exist" containerID="8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3" Oct 30 00:55:28.259674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.259595 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3"} err="failed to get container status \"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3\": rpc error: code = NotFound desc = could not find container \"8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3\": container with ID starting with 8ccfa9138ee55f76add72d695b6cde13c24851c6f2bde7b3bb6b4e316577f5d3 not found: ID does not exist" Oct 30 00:55:28.259674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.259618 5938 scope.go:122] "RemoveContainer" containerID="3c5f41319580f0e755d8dd2e86cfc6c5d4cc6fda2ea9b37d1a9aa5dc0aa0626f" Oct 30 00:55:28.261398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.261065 5938 generic.go:356] "Generic (PLEG): container finished" podID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerID="2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11" exitCode=0 Oct 30 00:55:28.261512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.261442 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" Oct 30 00:55:28.261586 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.261559 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" event={"ID":"944cdc20-56d4-41bf-9b47-4ee5e82b5d50","Type":"ContainerDied","Data":"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11"} Oct 30 00:55:28.261656 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.261613 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh" event={"ID":"944cdc20-56d4-41bf-9b47-4ee5e82b5d50","Type":"ContainerDied","Data":"a5a3b29391a11c13b36c5a3c865573eb8b3e3628403bc82c8be571b72f0b1e47"} Oct 30 00:55:28.265811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.265671 5938 generic.go:356] "Generic (PLEG): container finished" podID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerID="c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a" exitCode=0 Oct 30 00:55:28.265811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.265692 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerDied","Data":"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a"} Oct 30 00:55:28.265811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.265743 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" event={"ID":"cb14d4d8-31e4-4a02-925c-03fc3d350026","Type":"ContainerDied","Data":"5e3c25a035c4866d24f0254c21c9eff0c74fb2d199ceae8e7ec54e496e6a2025"} Oct 30 00:55:28.265811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.265803 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x" Oct 30 00:55:28.271363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.268037 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" event={"ID":"a34feee0-f121-4b7a-964a-aea2f35c1475","Type":"ContainerDied","Data":"b1ddaf1a559fd551f3021d1dec04f9d003a20c2ffff5d44885df47971f3f2cdc"} Oct 30 00:55:28.271363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.268145 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8" Oct 30 00:55:28.277023 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.276108 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:28.278437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.278418 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" event={"ID":"1b06c81b-19fa-4af1-9da1-b9a6dee53e53","Type":"ContainerDied","Data":"c301990bed963999c2a0ae91cfaedd1b6511128f26642264af5575769286ee7c"} Oct 30 00:55:28.279203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.278534 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5" Oct 30 00:55:28.290246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.290228 5938 scope.go:122] "RemoveContainer" containerID="ec254a66005b5205c80073a5a48af851e19c2067e551e35aed43a7674ca162c4" Oct 30 00:55:28.302432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.302401 5938 scope.go:122] "RemoveContainer" containerID="2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11" Oct 30 00:55:28.313739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.313714 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:55:28.315558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.315542 5938 scope.go:122] "RemoveContainer" containerID="2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11" Oct 30 00:55:28.315894 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:28.315856 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11\": container with ID starting with 2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11 not found: ID does not exist" containerID="2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11" Oct 30 00:55:28.315983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.315908 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11"} err="failed to get container status \"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11\": rpc error: code = NotFound desc = could not find container \"2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11\": container with ID starting with 2d6ee7c7e186e594fe7e61ddcb8cba0ac1bb268db406abf32f74196627081d11 not found: ID does not exist" Oct 30 00:55:28.315983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.315927 5938 scope.go:122] "RemoveContainer" containerID="c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a" Oct 30 00:55:28.325905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.324459 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-vqnx5"] Oct 30 00:55:28.326757 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.326742 5938 scope.go:122] "RemoveContainer" containerID="b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba" Oct 30 00:55:28.336296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.336276 5938 scope.go:122] "RemoveContainer" containerID="c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a" Oct 30 00:55:28.336582 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:28.336564 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a\": container with ID starting with c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a not found: ID does not exist" containerID="c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a" Oct 30 00:55:28.336663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.336588 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a"} err="failed to get container status \"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a\": rpc error: code = NotFound desc = could not find container \"c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a\": container with ID starting with c36c80ec603f1391dca479601977154d6b7641f2908893a8e4b10b76c014651a not found: ID does not exist" Oct 30 00:55:28.336663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.336603 5938 scope.go:122] "RemoveContainer" containerID="b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba" Oct 30 00:55:28.336846 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:28.336826 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba\": container with ID starting with b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba not found: ID does not exist" containerID="b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba" Oct 30 00:55:28.336919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.336852 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba"} err="failed to get container status \"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba\": rpc error: code = NotFound desc = could not find container \"b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba\": container with ID starting with b662b7471e682815cf58d955c33ce3132fed8067c867c1128e167cde0768f1ba not found: ID does not exist" Oct 30 00:55:28.336919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.336877 5938 scope.go:122] "RemoveContainer" containerID="25879a21ed10f6c99fbb02b655325f44f3dc0f6556ba957b02b0577fa4ad124f" Oct 30 00:55:28.346857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.346835 5938 scope.go:122] "RemoveContainer" containerID="97c206a165436ec46b8953ff403f35baa0071b422b17a95a2782a283574e06d9" Oct 30 00:55:28.356502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.356480 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:55:28.361174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.361155 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-825hf"] Oct 30 00:55:28.406645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.406620 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:55:28.412973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.412949 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-k46bb"] Oct 30 00:55:28.416533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.416506 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:28.534249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.529584 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:55:28.550376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.550305 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-prznh"] Oct 30 00:55:28.578123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.578088 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:55:28.584485 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.584455 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-cgv7x"] Oct 30 00:55:28.609058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.609032 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:55:28.612305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.612278 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-lmgj5"] Oct 30 00:55:28.648270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.648206 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:55:28.654772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:28.654750 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-nbcn8"] Oct 30 00:55:28.890687 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:28.890083 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:55:29.878993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.878967 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" path="/var/lib/kubelet/pods/1b06c81b-19fa-4af1-9da1-b9a6dee53e53/volumes" Oct 30 00:55:29.879475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.879458 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" path="/var/lib/kubelet/pods/213e2306-e63e-478a-9d7d-c1c5038343e3/volumes" Oct 30 00:55:29.879935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.879920 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" path="/var/lib/kubelet/pods/6a0b50c8-8bad-4637-89f0-a899b514d204/volumes" Oct 30 00:55:29.880725 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.880705 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" path="/var/lib/kubelet/pods/944cdc20-56d4-41bf-9b47-4ee5e82b5d50/volumes" Oct 30 00:55:29.881196 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.881181 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" path="/var/lib/kubelet/pods/a34feee0-f121-4b7a-964a-aea2f35c1475/volumes" Oct 30 00:55:29.881646 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.881631 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" path="/var/lib/kubelet/pods/cb14d4d8-31e4-4a02-925c-03fc3d350026/volumes" Oct 30 00:55:29.882458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.882438 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" path="/var/lib/kubelet/pods/e206ffd7-069f-4e1a-a4d0-31c5659f436c/volumes" Oct 30 00:55:29.883121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.883098 5938 scope.go:122] "RemoveContainer" containerID="f12bcd1b926d40aa916483f74b50c9a755d790d98a32523bc61d8b3d9ed2d492" Oct 30 00:55:29.919834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.919802 5938 scope.go:122] "RemoveContainer" containerID="807805be4605c00d38fd39d318072eb8e5dd5d0a13e3bb43e55ab45209fd0fee" Oct 30 00:55:29.933926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.933902 5938 scope.go:122] "RemoveContainer" containerID="4cab00f107e45e5911c9287699d50984c6ce3be31c3e1df0fcfaf77c5fa0bac5" Oct 30 00:55:29.943604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.943584 5938 scope.go:122] "RemoveContainer" containerID="23f6fd9670f12fb66a35859824e290d370719444e79aef147c8ab602f366494d" Oct 30 00:55:29.953778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.953753 5938 scope.go:122] "RemoveContainer" containerID="c81b43c2dd6e3ac30d6639329cfbcea3a638a2c1f166d5166577b163544fbbfa" Oct 30 00:55:29.962940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.962922 5938 scope.go:122] "RemoveContainer" containerID="21f74b6080c28e2ff4584a5f07e35372cd5104ea2616725e4110e53f8a64bcfc" Oct 30 00:55:29.972476 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.972459 5938 scope.go:122] "RemoveContainer" containerID="17370166a32ca472deee5218cfbf933bb87d57f00ba3345956ab5db40f0570ec" Oct 30 00:55:29.992033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:29.992009 5938 scope.go:122] "RemoveContainer" containerID="a4cddd656ff21e8c099fe4012a3e1644af527e1e8c17c856ab4c2e0461022c95" Oct 30 00:55:30.028762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.028611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:30.127955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.127930 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:55:30.276705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276681 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.276842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.276934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276856 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.276934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276886 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.276934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276914 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.277103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.276977 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.277103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277013 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.277218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277125 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:30.277218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277137 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:30.277218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277168 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.277218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277197 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\") pod \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\" (UID: \"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282\") " Oct 30 00:55:30.277436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277362 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:30.277627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277608 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:30.277758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.277678 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:30.278131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.278107 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.278280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.278134 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.278280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.278149 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.278280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.278162 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.278280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.278176 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.279576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.279543 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:30.279679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.279605 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:30.279679 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.279626 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "kube-api-access-zgjnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:30.279951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.279932 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo" pod "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" (UID: "d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:30.308033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.308006 5938 generic.go:356] "Generic (PLEG): container finished" podID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerID="27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9" exitCode=0 Oct 30 00:55:30.308121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.308058 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" Oct 30 00:55:30.308121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.308085 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerDied","Data":"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9"} Oct 30 00:55:30.308209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.308141 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4" event={"ID":"d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282","Type":"ContainerDied","Data":"4126973fc49f41a0ac8f3db864970332a9ca4e051c936c3d85d748389989f67f"} Oct 30 00:55:30.308209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.308161 5938 scope.go:122] "RemoveContainer" containerID="27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9" Oct 30 00:55:30.318427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.318413 5938 scope.go:122] "RemoveContainer" containerID="d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111" Oct 30 00:55:30.327410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.327387 5938 scope.go:122] "RemoveContainer" containerID="27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9" Oct 30 00:55:30.327675 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:30.327658 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9\": container with ID starting with 27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9 not found: ID does not exist" containerID="27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9" Oct 30 00:55:30.327745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.327682 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9"} err="failed to get container status \"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9\": rpc error: code = NotFound desc = could not find container \"27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9\": container with ID starting with 27f6e8457e7e289fa4763a470d8bdf1b315c0ba18ca00f7d832fbda730df03a9 not found: ID does not exist" Oct 30 00:55:30.327745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.327698 5938 scope.go:122] "RemoveContainer" containerID="d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111" Oct 30 00:55:30.327968 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:30.327952 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111\": container with ID starting with d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111 not found: ID does not exist" containerID="d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111" Oct 30 00:55:30.328020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.327973 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111"} err="failed to get container status \"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111\": rpc error: code = NotFound desc = could not find container \"d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111\": container with ID starting with d6456c75c873fa113a9ce40228e6e88c2cfdfedfa87c009c5a5de01e450e4111 not found: ID does not exist" Oct 30 00:55:30.359808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.359774 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:55:30.364425 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.364404 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fa555-openshift-default-574587c9ff-h28g4"] Oct 30 00:55:30.379154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.379132 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.379249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.379157 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgjnc\" (UniqueName: \"kubernetes.io/projected/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-kube-api-access-zgjnc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.379249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.379172 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:30.379249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:30.379187 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:31.158102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:31.158077 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:55:31.158498 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:31.158410 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:55:31.804767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:31.800198 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:31.879731 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:31.879697 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" path="/var/lib/kubelet/pods/d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282/volumes" Oct 30 00:55:32.356680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.356652 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8807-f8f452"] Oct 30 00:55:32.357375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357359 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" Oct 30 00:55:32.357375 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357375 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357384 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357390 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357412 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357417 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357426 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357431 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357439 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:32.357459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357446 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357471 5938 cpu_manager.go:414] "containerMap: removing container" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357476 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357485 5938 cpu_manager.go:414] "containerMap: removing container" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357489 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357496 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357500 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357512 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357516 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357534 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357538 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357546 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357550 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357645 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="944cdc20-56d4-41bf-9b47-4ee5e82b5d50" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357661 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="213e2306-e63e-478a-9d7d-c1c5038343e3" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357668 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357676 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357683 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357689 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ac6e7b-47e1-4eb6-96f2-abfc59e6b282" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357696 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34feee0-f121-4b7a-964a-aea2f35c1475" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357701 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e206ffd7-069f-4e1a-a4d0-31c5659f436c" containerName="istio-proxy" Oct 30 00:55:32.357726 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.357708 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0b50c8-8bad-4637-89f0-a899b514d204" containerName="istio-proxy" Oct 30 00:55:32.364823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.364795 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:32.377221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.376216 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8807-f8f452"] Oct 30 00:55:32.383105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.383072 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\"" Oct 30 00:55:32.509827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.509792 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkclq\" (UniqueName: \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\") pod \"test-curl-8807-f8f452\" (UID: \"1c081c8c-9a41-4f87-86d5-5480170ec0a6\") " pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:32.617961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.616101 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkclq\" (UniqueName: \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\") pod \"test-curl-8807-f8f452\" (UID: \"1c081c8c-9a41-4f87-86d5-5480170ec0a6\") " pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:32.659887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.653070 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkclq\" (UniqueName: \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\") pod \"test-curl-8807-f8f452\" (UID: \"1c081c8c-9a41-4f87-86d5-5480170ec0a6\") " pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:32.695884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:32.695114 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:33.051977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.051948 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8807-f8f452"] Oct 30 00:55:33.053351 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:33.053320 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c081c8c_9a41_4f87_86d5_5480170ec0a6.slice/crio-0232754b4ad0ada01b56adddc303b23a394a38c9dafb4d850d91f51f6e756d52 WatchSource:0}: Error finding container 0232754b4ad0ada01b56adddc303b23a394a38c9dafb4d850d91f51f6e756d52: Status 404 returned error can't find the container with id 0232754b4ad0ada01b56adddc303b23a394a38c9dafb4d850d91f51f6e756d52 Oct 30 00:55:33.093086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.093052 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" podUID="43168dae-5f77-4d46-ba9a-098b000abea5" containerName="payload-processing" containerID="cri-o://4fc77ccf8846018d0701a43f681c19ef6de9474d35a85ba249f86e52a575e76a" gracePeriod=25 Oct 30 00:55:33.125722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.125690 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" podUID="055dfa30-388c-463a-88c2-e3022034c5cb" containerName="payload-pre-processing" containerID="cri-o://143be4f7fb41a74084dd3f8f850f7a41f73f323f10c4662107536ee50513e33c" gracePeriod=25 Oct 30 00:55:33.343005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.341155 5938 generic.go:356] "Generic (PLEG): container finished" podID="055dfa30-388c-463a-88c2-e3022034c5cb" containerID="143be4f7fb41a74084dd3f8f850f7a41f73f323f10c4662107536ee50513e33c" exitCode=0 Oct 30 00:55:33.343005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.341224 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" event={"ID":"055dfa30-388c-463a-88c2-e3022034c5cb","Type":"ContainerDied","Data":"143be4f7fb41a74084dd3f8f850f7a41f73f323f10c4662107536ee50513e33c"} Oct 30 00:55:33.343879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.343332 5938 generic.go:356] "Generic (PLEG): container finished" podID="43168dae-5f77-4d46-ba9a-098b000abea5" containerID="4fc77ccf8846018d0701a43f681c19ef6de9474d35a85ba249f86e52a575e76a" exitCode=0 Oct 30 00:55:33.343879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.343473 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" event={"ID":"43168dae-5f77-4d46-ba9a-098b000abea5","Type":"ContainerDied","Data":"4fc77ccf8846018d0701a43f681c19ef6de9474d35a85ba249f86e52a575e76a"} Oct 30 00:55:33.346908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.346287 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8807-f8f452" event={"ID":"1c081c8c-9a41-4f87-86d5-5480170ec0a6","Type":"ContainerStarted","Data":"8dff406254c14996e5d615ed7e7940e39cd3283be95d30242e5e09f904c6d3e6"} Oct 30 00:55:33.346908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.346325 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8807-f8f452" event={"ID":"1c081c8c-9a41-4f87-86d5-5480170ec0a6","Type":"ContainerStarted","Data":"0232754b4ad0ada01b56adddc303b23a394a38c9dafb4d850d91f51f6e756d52"} Oct 30 00:55:33.381302 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.381271 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:33.413004 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.411854 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8807-f8f452" podStartSLOduration=1.411829484 podStartE2EDuration="1.411829484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:33.39690161 +0000 UTC m=+1385.637136724" watchObservedRunningTime="2026-10-30 00:55:33.411829484 +0000 UTC m=+1385.652064597" Oct 30 00:55:33.466805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.466747 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:55:33.482358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.482320 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:55:33.628938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.627253 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume" pod "43168dae-5f77-4d46-ba9a-098b000abea5" (UID: "43168dae-5f77-4d46-ba9a-098b000abea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:33.628938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.628077 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\") pod \"43168dae-5f77-4d46-ba9a-098b000abea5\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " Oct 30 00:55:33.628938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.628174 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\" (UniqueName: \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\") pod \"055dfa30-388c-463a-88c2-e3022034c5cb\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " Oct 30 00:55:33.628938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.628510 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\" (UniqueName: \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\") pod \"43168dae-5f77-4d46-ba9a-098b000abea5\" (UID: \"43168dae-5f77-4d46-ba9a-098b000abea5\") " Oct 30 00:55:33.628938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.628551 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\") pod \"055dfa30-388c-463a-88c2-e3022034c5cb\" (UID: \"055dfa30-388c-463a-88c2-e3022034c5cb\") " Oct 30 00:55:33.658918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.631204 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43168dae-5f77-4d46-ba9a-098b000abea5-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:33.658918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.631575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume" pod "055dfa30-388c-463a-88c2-e3022034c5cb" (UID: "055dfa30-388c-463a-88c2-e3022034c5cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:33.658918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.635539 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm" pod "055dfa30-388c-463a-88c2-e3022034c5cb" (UID: "055dfa30-388c-463a-88c2-e3022034c5cb"). InnerVolumeSpecName "kube-api-access-x5ncm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:33.658918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.636122 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z" pod "43168dae-5f77-4d46-ba9a-098b000abea5" (UID: "43168dae-5f77-4d46-ba9a-098b000abea5"). InnerVolumeSpecName "kube-api-access-bbn4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:33.733932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.733898 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbn4z\" (UniqueName: \"kubernetes.io/projected/43168dae-5f77-4d46-ba9a-098b000abea5-kube-api-access-bbn4z\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:33.733932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.733934 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055dfa30-388c-463a-88c2-e3022034c5cb-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:33.734120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:33.733949 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ncm\" (UniqueName: \"kubernetes.io/projected/055dfa30-388c-463a-88c2-e3022034c5cb-kube-api-access-x5ncm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:34.355560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.355529 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" Oct 30 00:55:34.355762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.355566 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x" event={"ID":"055dfa30-388c-463a-88c2-e3022034c5cb","Type":"ContainerDied","Data":"c8868287bd4b61f4275c97b217057e4f21d52e78cac16cb5858fdf7280bd51ce"} Oct 30 00:55:34.355762 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.355643 5938 scope.go:122] "RemoveContainer" containerID="143be4f7fb41a74084dd3f8f850f7a41f73f323f10c4662107536ee50513e33c" Oct 30 00:55:34.358020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.357728 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" Oct 30 00:55:34.358020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.357791 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw" event={"ID":"43168dae-5f77-4d46-ba9a-098b000abea5","Type":"ContainerDied","Data":"ba815f5be8b967d079bd928b0c89ae4af3b0891cea231d27205a59f01075aa0a"} Oct 30 00:55:34.367132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.367115 5938 scope.go:122] "RemoveContainer" containerID="4fc77ccf8846018d0701a43f681c19ef6de9474d35a85ba249f86e52a575e76a" Oct 30 00:55:34.407047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.407024 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:55:34.420745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.420721 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d788ba-5fcb7b4fc8-n8fvw"] Oct 30 00:55:34.443927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.443287 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:55:34.455314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.455285 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d788ba-5ff46bcf95-hlr4x"] Oct 30 00:55:34.533508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.533475 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8807-f8f452"] Oct 30 00:55:34.543045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.543020 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8807-f8f452"] Oct 30 00:55:34.791310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.791208 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4"] Oct 30 00:55:34.792787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792763 5938 cpu_manager.go:414] "containerMap: removing container" podUID="055dfa30-388c-463a-88c2-e3022034c5cb" containerName="payload-pre-processing" Oct 30 00:55:34.792787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792784 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055dfa30-388c-463a-88c2-e3022034c5cb" containerName="payload-pre-processing" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792813 5938 cpu_manager.go:414] "containerMap: removing container" podUID="43168dae-5f77-4d46-ba9a-098b000abea5" containerName="payload-processing" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792819 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43168dae-5f77-4d46-ba9a-098b000abea5" containerName="payload-processing" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792836 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" containerName="test-curl-8807-f8f452" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.792841 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" containerName="test-curl-8807-f8f452" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.793005 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb14d4d8-31e4-4a02-925c-03fc3d350026" containerName="istio-proxy" Oct 30 00:55:34.793018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.793021 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b06c81b-19fa-4af1-9da1-b9a6dee53e53" containerName="istio-proxy" Oct 30 00:55:34.793335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.793032 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="055dfa30-388c-463a-88c2-e3022034c5cb" containerName="payload-pre-processing" Oct 30 00:55:34.793335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.793045 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="43168dae-5f77-4d46-ba9a-098b000abea5" containerName="payload-processing" Oct 30 00:55:34.793335 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.793053 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" containerName="test-curl-8807-f8f452" Oct 30 00:55:34.800185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.800149 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:34.813617 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.812701 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4"] Oct 30 00:55:34.815431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.815411 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 00:55:34.818542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.816269 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 00:55:34.964085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.964050 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:34.964624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.964601 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:34.964790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:34.964776 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45sqx\" (UniqueName: \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.066830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.066727 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.066830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.066797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.066830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.066828 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45sqx\" (UniqueName: \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.067663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.067634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.069392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.069365 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.090263 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.090224 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45sqx\" (UniqueName: \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\") pod \"maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.134958 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.134928 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:35.373381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.373340 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8807-f8f452" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" containerName="test-curl-8807-f8f452" containerID="cri-o://8dff406254c14996e5d615ed7e7940e39cd3283be95d30242e5e09f904c6d3e6" gracePeriod=2 Oct 30 00:55:35.386941 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:35.386910 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:35.458675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.458647 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4"] Oct 30 00:55:35.461188 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:35.461161 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0b10d1_eb58_44c5_a951_320fd5d4859e.slice/crio-cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea WatchSource:0}: Error finding container cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea: Status 404 returned error can't find the container with id cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea Oct 30 00:55:35.705075 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.705049 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:35.886110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.886073 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055dfa30-388c-463a-88c2-e3022034c5cb" path="/var/lib/kubelet/pods/055dfa30-388c-463a-88c2-e3022034c5cb/volumes" Oct 30 00:55:35.886708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:35.886688 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43168dae-5f77-4d46-ba9a-098b000abea5" path="/var/lib/kubelet/pods/43168dae-5f77-4d46-ba9a-098b000abea5/volumes" Oct 30 00:55:36.106884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.106416 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:55:36.154885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.154750 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:55:36.308716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.307888 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:36.532071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.531324 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:36.736241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.736199 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" event={"ID":"7a0b10d1-eb58-44c5-a951-320fd5d4859e","Type":"ContainerStarted","Data":"a49c298d1bb73f806330ca6dbedfca87cb9675ea0c8a5b935bda336a9a1f1338"} Oct 30 00:55:36.736501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.736468 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" event={"ID":"7a0b10d1-eb58-44c5-a951-320fd5d4859e","Type":"ContainerStarted","Data":"cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea"} Oct 30 00:55:36.828035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.827907 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:55:36.847906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:36.830697 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" containerID="cri-o://bee3b86c676b66d70fcb1c878ab6d4a9b5854e5b4fff3d33a03ca686c89195ae" gracePeriod=30 Oct 30 00:55:37.475909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:37.474774 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:37.933086 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:37.933050 5938 generic.go:356] "Generic (PLEG): container finished" podID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" containerID="8dff406254c14996e5d615ed7e7940e39cd3283be95d30242e5e09f904c6d3e6" exitCode=137 Oct 30 00:55:37.935922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:37.935519 5938 generic.go:356] "Generic (PLEG): container finished" podID="7a0b10d1-eb58-44c5-a951-320fd5d4859e" containerID="a49c298d1bb73f806330ca6dbedfca87cb9675ea0c8a5b935bda336a9a1f1338" exitCode=0 Oct 30 00:55:37.944693 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:37.944670 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" event={"ID":"7a0b10d1-eb58-44c5-a951-320fd5d4859e","Type":"ContainerDied","Data":"a49c298d1bb73f806330ca6dbedfca87cb9675ea0c8a5b935bda336a9a1f1338"} Oct 30 00:55:37.971400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:37.970009 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:37.984035 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:37.983999 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:38.029232 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.028436 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:38.068911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.067848 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\" (UniqueName: \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\") pod \"1c081c8c-9a41-4f87-86d5-5480170ec0a6\" (UID: \"1c081c8c-9a41-4f87-86d5-5480170ec0a6\") " Oct 30 00:55:38.070274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.070243 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq" pod "1c081c8c-9a41-4f87-86d5-5480170ec0a6" (UID: "1c081c8c-9a41-4f87-86d5-5480170ec0a6"). InnerVolumeSpecName "kube-api-access-pkclq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:38.120573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.120291 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:38.170763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.170735 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkclq\" (UniqueName: \"kubernetes.io/projected/1c081c8c-9a41-4f87-86d5-5480170ec0a6-kube-api-access-pkclq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.171107 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:38.171080 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:38.271716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.271686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\") pod \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " Oct 30 00:55:38.271915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.271745 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\") pod \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " Oct 30 00:55:38.271915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.271795 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\" (UniqueName: \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\") pod \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\" (UID: \"7a0b10d1-eb58-44c5-a951-320fd5d4859e\") " Oct 30 00:55:38.272105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.272080 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca" pod "7a0b10d1-eb58-44c5-a951-320fd5d4859e" (UID: "7a0b10d1-eb58-44c5-a951-320fd5d4859e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:38.272738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.272708 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.273979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.273942 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx" pod "7a0b10d1-eb58-44c5-a951-320fd5d4859e" (UID: "7a0b10d1-eb58-44c5-a951-320fd5d4859e"). InnerVolumeSpecName "kube-api-access-45sqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:38.274068 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.273980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca" pod "7a0b10d1-eb58-44c5-a951-320fd5d4859e" (UID: "7a0b10d1-eb58-44c5-a951-320fd5d4859e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:55:38.373664 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.373644 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0b10d1-eb58-44c5-a951-320fd5d4859e-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.373759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.373666 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45sqx\" (UniqueName: \"kubernetes.io/projected/7a0b10d1-eb58-44c5-a951-320fd5d4859e-kube-api-access-45sqx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:38.944451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.944419 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:38.945218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.944463 5938 scope.go:122] "RemoveContainer" containerID="8dff406254c14996e5d615ed7e7940e39cd3283be95d30242e5e09f904c6d3e6" Oct 30 00:55:38.947650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.946613 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" event={"ID":"7a0b10d1-eb58-44c5-a951-320fd5d4859e","Type":"ContainerDied","Data":"cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea"} Oct 30 00:55:38.947650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.946671 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd61d87257c348951060973b9c70a2e3d9aece1f74d0399308f925fd3401feea" Oct 30 00:55:38.947650 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:38.946691 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4" Oct 30 00:55:38.957442 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:38.957421 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:38.973274 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:38.970229 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:38.997129 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:38.997100 5938 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8807-f8f452\" is forbidden: User \"system:node:ip-10-0-131-130.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-130.ec2.internal' and this object" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" pod="opendatahub/test-curl-8807-f8f452" Oct 30 00:55:39.874246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:39.874209 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:39.879964 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:39.879934 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c081c8c-9a41-4f87-86d5-5480170ec0a6" path="/var/lib/kubelet/pods/1c081c8c-9a41-4f87-86d5-5480170ec0a6/volumes" Oct 30 00:55:41.107184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.107146 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" podUID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" containerName="payload-processing" containerID="cri-o://f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059" gracePeriod=25 Oct 30 00:55:41.155576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.155427 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" podUID="bba98624-b644-4722-894c-03259b6ed99c" containerName="payload-pre-processing" containerID="cri-o://e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23" gracePeriod=25 Oct 30 00:55:41.392489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.392297 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:41.495030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.494993 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:55:41.498660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.498641 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:55:41.618016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.617982 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\") pod \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " Oct 30 00:55:41.618172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.618094 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\") pod \"bba98624-b644-4722-894c-03259b6ed99c\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " Oct 30 00:55:41.618235 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.618191 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\" (UniqueName: \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\") pod \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\" (UID: \"d4d0f7e1-0bf6-4673-92dc-b078cddba4df\") " Oct 30 00:55:41.618386 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.618370 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\" (UniqueName: \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\") pod \"bba98624-b644-4722-894c-03259b6ed99c\" (UID: \"bba98624-b644-4722-894c-03259b6ed99c\") " Oct 30 00:55:41.618465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.618390 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume" pod "d4d0f7e1-0bf6-4673-92dc-b078cddba4df" (UID: "d4d0f7e1-0bf6-4673-92dc-b078cddba4df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:41.618465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.618425 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume" pod "bba98624-b644-4722-894c-03259b6ed99c" (UID: "bba98624-b644-4722-894c-03259b6ed99c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:41.619047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.619026 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.619150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.619048 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba98624-b644-4722-894c-03259b6ed99c-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.620318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.620296 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d" pod "d4d0f7e1-0bf6-4673-92dc-b078cddba4df" (UID: "d4d0f7e1-0bf6-4673-92dc-b078cddba4df"). InnerVolumeSpecName "kube-api-access-h6m5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:41.620407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.620338 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4" pod "bba98624-b644-4722-894c-03259b6ed99c" (UID: "bba98624-b644-4722-894c-03259b6ed99c"). InnerVolumeSpecName "kube-api-access-9ztb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:41.721007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.720944 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ztb4\" (UniqueName: \"kubernetes.io/projected/bba98624-b644-4722-894c-03259b6ed99c-kube-api-access-9ztb4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.721007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.720968 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6m5d\" (UniqueName: \"kubernetes.io/projected/d4d0f7e1-0bf6-4673-92dc-b078cddba4df-kube-api-access-h6m5d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:41.974743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.974658 5938 generic.go:356] "Generic (PLEG): container finished" podID="bba98624-b644-4722-894c-03259b6ed99c" containerID="e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23" exitCode=0 Oct 30 00:55:41.974743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.974730 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" Oct 30 00:55:41.975026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.974745 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" event={"ID":"bba98624-b644-4722-894c-03259b6ed99c","Type":"ContainerDied","Data":"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23"} Oct 30 00:55:41.975026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.974833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv" event={"ID":"bba98624-b644-4722-894c-03259b6ed99c","Type":"ContainerDied","Data":"db735688a396d2b0e731c32d3b527f6cd107c06ce22e2abdd907725590d2c8e9"} Oct 30 00:55:41.975026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.974910 5938 scope.go:122] "RemoveContainer" containerID="e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23" Oct 30 00:55:41.978357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.978334 5938 generic.go:356] "Generic (PLEG): container finished" podID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerID="bee3b86c676b66d70fcb1c878ab6d4a9b5854e5b4fff3d33a03ca686c89195ae" exitCode=0 Oct 30 00:55:41.978491 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.978413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" event={"ID":"b24a1159-1d8d-48dd-bbac-78f509c62976","Type":"ContainerDied","Data":"bee3b86c676b66d70fcb1c878ab6d4a9b5854e5b4fff3d33a03ca686c89195ae"} Oct 30 00:55:41.980115 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.980092 5938 generic.go:356] "Generic (PLEG): container finished" podID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" containerID="f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059" exitCode=0 Oct 30 00:55:41.980293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.980275 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" event={"ID":"d4d0f7e1-0bf6-4673-92dc-b078cddba4df","Type":"ContainerDied","Data":"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059"} Oct 30 00:55:41.980367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.980319 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" event={"ID":"d4d0f7e1-0bf6-4673-92dc-b078cddba4df","Type":"ContainerDied","Data":"53525f0d91fc5943775e7ad628d287e0508cf8af1816fa65f5f7a99a7255bdb6"} Oct 30 00:55:41.980427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.980398 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2" Oct 30 00:55:41.991491 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.991348 5938 scope.go:122] "RemoveContainer" containerID="e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23" Oct 30 00:55:41.991716 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:41.991693 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23\": container with ID starting with e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23 not found: ID does not exist" containerID="e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23" Oct 30 00:55:41.991805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.991723 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23"} err="failed to get container status \"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23\": rpc error: code = NotFound desc = could not find container \"e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23\": container with ID starting with e3435daebefeedc172430341737ddff372f4dd7bc25ec2dd2aab39dddf236b23 not found: ID does not exist" Oct 30 00:55:41.991805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:41.991741 5938 scope.go:122] "RemoveContainer" containerID="f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059" Oct 30 00:55:42.008392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.008373 5938 scope.go:122] "RemoveContainer" containerID="f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059" Oct 30 00:55:42.008709 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:42.008682 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059\": container with ID starting with f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059 not found: ID does not exist" containerID="f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059" Oct 30 00:55:42.008776 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.008718 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059"} err="failed to get container status \"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059\": rpc error: code = NotFound desc = could not find container \"f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059\": container with ID starting with f8d0f3350332687906d8ae6a2d452d57e618f0c6e47c3f4a27738af91e108059 not found: ID does not exist" Oct 30 00:55:42.034644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.034611 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:55:42.048012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.047983 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bdc256-66b6b9d4f6-6sjc2"] Oct 30 00:55:42.079493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.079436 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:55:42.086104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.086059 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bdc256-7559f49547-94jpv"] Oct 30 00:55:42.104100 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.104080 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:55:42.228710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.228710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228693 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228747 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228794 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228816 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228951 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.228977 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229047 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229118 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:42.229568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229226 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\") pod \"b24a1159-1d8d-48dd-bbac-78f509c62976\" (UID: \"b24a1159-1d8d-48dd-bbac-78f509c62976\") " Oct 30 00:55:42.229568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:42.229568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229332 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:42.229568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229356 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:42.229568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229387 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:42.229921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229826 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.229921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229857 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.229921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229888 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24a1159-1d8d-48dd-bbac-78f509c62976-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.229921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229903 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.229921 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.229916 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.231309 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.231283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "kube-api-access-cgngv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:42.231469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.231439 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:42.231469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.231457 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:55:42.231595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.231479 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy" pod "b24a1159-1d8d-48dd-bbac-78f509c62976" (UID: "b24a1159-1d8d-48dd-bbac-78f509c62976"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:55:42.331430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.331400 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.331430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.331425 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.331430 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.331435 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgngv\" (UniqueName: \"kubernetes.io/projected/b24a1159-1d8d-48dd-bbac-78f509c62976-kube-api-access-cgngv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.331640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.331444 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24a1159-1d8d-48dd-bbac-78f509c62976-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:42.991285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.991251 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" event={"ID":"b24a1159-1d8d-48dd-bbac-78f509c62976","Type":"ContainerDied","Data":"19ef8b772d245a172774f7d2f033d012a583224c01bfae5f9482e728311d5273"} Oct 30 00:55:42.991285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.991283 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx" Oct 30 00:55:42.991532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:42.991311 5938 scope.go:122] "RemoveContainer" containerID="bee3b86c676b66d70fcb1c878ab6d4a9b5854e5b4fff3d33a03ca686c89195ae" Oct 30 00:55:43.054362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.054333 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:55:43.065790 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.065763 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-t6pxx"] Oct 30 00:55:43.186128 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.186090 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:43.877846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.877803 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" path="/var/lib/kubelet/pods/b24a1159-1d8d-48dd-bbac-78f509c62976/volumes" Oct 30 00:55:43.878348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.878327 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba98624-b644-4722-894c-03259b6ed99c" path="/var/lib/kubelet/pods/bba98624-b644-4722-894c-03259b6ed99c/volumes" Oct 30 00:55:43.878847 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:43.878824 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" path="/var/lib/kubelet/pods/d4d0f7e1-0bf6-4673-92dc-b078cddba4df/volumes" Oct 30 00:55:44.010301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.010267 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 00:55:44.010972 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.010954 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" containerName="payload-processing" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.010973 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" containerName="payload-processing" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.010997 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bba98624-b644-4722-894c-03259b6ed99c" containerName="payload-pre-processing" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011004 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba98624-b644-4722-894c-03259b6ed99c" containerName="payload-pre-processing" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011040 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011048 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011093 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7a0b10d1-eb58-44c5-a951-320fd5d4859e" containerName="revoke-keys" Oct 30 00:55:44.011103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011100 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0b10d1-eb58-44c5-a951-320fd5d4859e" containerName="revoke-keys" Oct 30 00:55:44.011440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011254 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0b10d1-eb58-44c5-a951-320fd5d4859e" containerName="revoke-keys" Oct 30 00:55:44.011440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011273 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d0f7e1-0bf6-4673-92dc-b078cddba4df" containerName="payload-processing" Oct 30 00:55:44.011440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011285 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba98624-b644-4722-894c-03259b6ed99c" containerName="payload-pre-processing" Oct 30 00:55:44.011440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.011299 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24a1159-1d8d-48dd-bbac-78f509c62976" containerName="istio-proxy" Oct 30 00:55:44.018887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.017668 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.035493 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.035468 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 00:55:44.043306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.043285 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 00:55:44.043593 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.043567 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 00:55:44.043723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.043698 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k4cpq\"" Oct 30 00:55:44.043814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.043747 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 00:55:44.154238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154173 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154208 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154405 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrd7\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154558 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.154620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.154606 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256196 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256253 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrd7\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256290 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256377 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256402 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.256998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.256971 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.258910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.258856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.258910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.258904 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.259042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.258967 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.259042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.258992 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.279135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.279114 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.279379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.279361 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrd7\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5gbdq\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.333477 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.333454 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:44.652433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.652403 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 00:55:44.654119 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:44.654093 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefcb6bf_7322_4db7_885d_b0f0a75866f3.slice/crio-c929fd4a8213ea020954f3fadcc6db5a9871565e6ca1baee390257f4a1890f0f WatchSource:0}: Error finding container c929fd4a8213ea020954f3fadcc6db5a9871565e6ca1baee390257f4a1890f0f: Status 404 returned error can't find the container with id c929fd4a8213ea020954f3fadcc6db5a9871565e6ca1baee390257f4a1890f0f Oct 30 00:55:44.798007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:44.797972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:45.009519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:45.009433 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" event={"ID":"defcb6bf-7322-4db7-885d-b0f0a75866f3","Type":"ContainerStarted","Data":"c929fd4a8213ea020954f3fadcc6db5a9871565e6ca1baee390257f4a1890f0f"} Oct 30 00:55:45.358754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:45.358715 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:55:45.503331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:45.503283 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:45.648702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:45.648612 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:46.759393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:46.759354 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:47.417018 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:47.416979 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:47.417138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:47.417057 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:48.070468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.070423 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" event={"ID":"defcb6bf-7322-4db7-885d-b0f0a75866f3","Type":"ContainerStarted","Data":"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1"} Oct 30 00:55:48.071000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.070605 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:48.072788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.072714 5938 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5gbdq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:55:48.072788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.072772 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:55:48.152889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.135040 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" podStartSLOduration=2.375741928 podStartE2EDuration="5.135009735s" totalImagesPullingTime="2.759267807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:48.127515866 +0000 UTC m=+1400.367750981" watchObservedRunningTime="2026-10-30 00:55:48.135009735 +0000 UTC m=+1400.375244849" Oct 30 00:55:48.262886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.253418 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:55:48.312885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.298448 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.312885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.298355 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:55:48.320920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.313502 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fa8dd0-serving-cert\"" Oct 30 00:55:48.402453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.391054 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.402453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.391141 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8thd\" (UniqueName: \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.402453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.391262 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.402453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.391405 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.402453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.393375 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.496346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.496229 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.496690 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.496562 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8thd\" (UniqueName: \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.496890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.496804 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.497272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.497149 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.497975 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:48.497739 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-serving-cert: secret "maas-api-e2e-api-a-fa8dd0-serving-cert" not found Oct 30 00:55:48.497975 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:48.497824 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls podName:6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9 nodeName:}" failed. No retries permitted until 2026-10-30 00:55:48.997802083 +0000 UTC m=+1401.238037174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls") pod "maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" (UID: "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9") : secret "maas-api-e2e-api-a-fa8dd0-serving-cert" not found Oct 30 00:55:48.499770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.499750 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.507564 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.505118 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.526323 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.526256 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8thd\" (UniqueName: \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:48.613783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.613751 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:48.810944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:48.810912 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.004007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:49.003974 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:49.006484 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:49.006453 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") pod \"maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:49.078936 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:49.078914 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 00:55:49.232234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:49.232167 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:49.587750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:49.587720 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:55:49.588202 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:49.588090 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2cbe62_d2d8_45cc_94bd_4c9e041a34f9.slice/crio-23b6f66c1a81d9b8b32a532f49970bf03363c2b3f374d63b0ea079c00df6965f WatchSource:0}: Error finding container 23b6f66c1a81d9b8b32a532f49970bf03363c2b3f374d63b0ea079c00df6965f: Status 404 returned error can't find the container with id 23b6f66c1a81d9b8b32a532f49970bf03363c2b3f374d63b0ea079c00df6965f Oct 30 00:55:50.094902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.094855 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" event={"ID":"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9","Type":"ContainerStarted","Data":"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de"} Oct 30 00:55:50.095353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.095307 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:50.095353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.095333 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" event={"ID":"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9","Type":"ContainerStarted","Data":"23b6f66c1a81d9b8b32a532f49970bf03363c2b3f374d63b0ea079c00df6965f"} Oct 30 00:55:50.142741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.142684 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" podStartSLOduration=1.771990013 podStartE2EDuration="2.142666624s" totalImagesPullingTime="370.676611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:50.13733485 +0000 UTC m=+1402.377569966" watchObservedRunningTime="2026-10-30 00:55:50.142666624 +0000 UTC m=+1402.382901738" Oct 30 00:55:50.359488 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.359431 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" podUID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" containerName="payload-pre-processing" containerID="cri-o://452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc" gracePeriod=25 Oct 30 00:55:50.514686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.514656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.665579 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.665552 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:55:50.843712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.843683 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\") pod \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " Oct 30 00:55:50.843904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.843890 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\" (UniqueName: \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\") pod \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\" (UID: \"d3f9d5cb-58df-4822-8ce1-cd506007c9fb\") " Oct 30 00:55:50.844082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.844062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume" pod "d3f9d5cb-58df-4822-8ce1-cd506007c9fb" (UID: "d3f9d5cb-58df-4822-8ce1-cd506007c9fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:55:50.844343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.844323 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:50.845900 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.845858 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn" pod "d3f9d5cb-58df-4822-8ce1-cd506007c9fb" (UID: "d3f9d5cb-58df-4822-8ce1-cd506007c9fb"). InnerVolumeSpecName "kube-api-access-xnkpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:55:50.945996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:50.945942 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkpn\" (UniqueName: \"kubernetes.io/projected/d3f9d5cb-58df-4822-8ce1-cd506007c9fb-kube-api-access-xnkpn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:55:51.105475 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.104931 5938 generic.go:356] "Generic (PLEG): container finished" podID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" containerID="452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc" exitCode=0 Oct 30 00:55:51.105944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.105742 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" Oct 30 00:55:51.107032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.106986 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" event={"ID":"d3f9d5cb-58df-4822-8ce1-cd506007c9fb","Type":"ContainerDied","Data":"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc"} Oct 30 00:55:51.107132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.107053 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4" event={"ID":"d3f9d5cb-58df-4822-8ce1-cd506007c9fb","Type":"ContainerDied","Data":"6a5f72951882b2156cb38b7530c55cfd53950ab699d5f914ff70b7266aeb4a50"} Oct 30 00:55:51.107132 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.107087 5938 scope.go:122] "RemoveContainer" containerID="452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc" Oct 30 00:55:51.120287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.120270 5938 scope.go:122] "RemoveContainer" containerID="452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc" Oct 30 00:55:51.120586 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:51.120566 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc\": container with ID starting with 452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc not found: ID does not exist" containerID="452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc" Oct 30 00:55:51.120658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.120596 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc"} err="failed to get container status \"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc\": rpc error: code = NotFound desc = could not find container \"452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc\": container with ID starting with 452b83d9a2ab254168316c9cc102a17fd2f1d79d326cef7e1d4776f25bb613bc not found: ID does not exist" Oct 30 00:55:51.157934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.157910 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:55:51.161984 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.161955 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-bpxv4"] Oct 30 00:55:51.300357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.300250 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.733162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.733112 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.877588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:51.877559 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" path="/var/lib/kubelet/pods/d3f9d5cb-58df-4822-8ce1-cd506007c9fb/volumes" Oct 30 00:55:52.139982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.139940 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 00:55:52.140701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.140680 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" containerName="payload-pre-processing" Oct 30 00:55:52.140701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.140698 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" containerName="payload-pre-processing" Oct 30 00:55:52.140975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.140961 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f9d5cb-58df-4822-8ce1-cd506007c9fb" containerName="payload-pre-processing" Oct 30 00:55:52.145908 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.145885 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.175940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.175910 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b3bdf9-openshift-default-dockercfg-fqtc6\"" Oct 30 00:55:52.182337 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.182309 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 00:55:52.228960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.228929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:52.263622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263594 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.263741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263634 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.263741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.263830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263804 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.263966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263950 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.264007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.263989 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.264150 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.264137 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.264203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.264193 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.264256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.264241 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5kp\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365179 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365218 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365259 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365392 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365430 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365555 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365658 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5kp\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.365807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365789 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.366059 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.365998 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.366166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.366058 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.366166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.366125 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.366166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.366130 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.368060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.368033 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.368434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.368413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.374289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.374266 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:52.423934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.416443 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.442286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.429541 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5kp\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\") pod \"e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.470890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.466561 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:52.571020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.570969 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:52.871968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.871942 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 00:55:52.872447 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:52.872425 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254296d8_2085_4cc9_b2dd_63d78b0643c8.slice/crio-27d76376d9b4e59e176de84ac1f458f87b4aaeff383b58185099a2df225417e1 WatchSource:0}: Error finding container 27d76376d9b4e59e176de84ac1f458f87b4aaeff383b58185099a2df225417e1: Status 404 returned error can't find the container with id 27d76376d9b4e59e176de84ac1f458f87b4aaeff383b58185099a2df225417e1 Oct 30 00:55:52.875145 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.875100 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:52.875218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.875182 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:52.875267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:52.875213 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:55:53.134701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:53.134601 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerStarted","Data":"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753"} Oct 30 00:55:53.134701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:53.134700 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerStarted","Data":"27d76376d9b4e59e176de84ac1f458f87b4aaeff383b58185099a2df225417e1"} Oct 30 00:55:53.191155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:53.191105 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podStartSLOduration=1.191092513 podStartE2EDuration="1.191092513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:53.177520524 +0000 UTC m=+1405.417755637" watchObservedRunningTime="2026-10-30 00:55:53.191092513 +0000 UTC m=+1405.431327626" Oct 30 00:55:53.467258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:53.467176 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:53.472661 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:53.472639 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:54.021057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:54.020660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.142665 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:54.142635 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:54.143500 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:54.143480 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:55:55.454288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:55.454261 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.609427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:55.609360 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.808642 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:55.806191 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.111721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.111695 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:55:56.270680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.270646 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.356176 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.356142 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4"] Oct 30 00:55:56.359401 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.359377 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d7a6db25-rmwd4"] Oct 30 00:55:56.422044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.421974 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.560526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.560489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:56.893353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:56.893321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.323703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.323621 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.466440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.466410 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 00:55:57.470985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.470958 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.471159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.471045 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.487895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.487845 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 00:55:57.495317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.495292 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b3bdf9\"" Oct 30 00:55:57.495317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.495300 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b3bdf9-dockercfg-zqxbd\"" Oct 30 00:55:57.572883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.572169 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.572883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.572319 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25s9w\" (UniqueName: \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.611028 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.610105 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 00:55:57.629399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.627460 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.638316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.638000 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 00:55:57.675841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.675807 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnvg\" (UniqueName: \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.676194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.676169 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.676330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.676311 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25s9w\" (UniqueName: \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.676398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.676350 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.677433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.677408 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.698741 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.698720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25s9w\" (UniqueName: \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\") pod \"payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.778197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.778162 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.778380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.778355 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnvg\" (UniqueName: \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.780800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.780267 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.786331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.786304 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:57.824606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.824573 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnvg\" (UniqueName: \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\") pod \"payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:57.879345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.879271 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0b10d1-eb58-44c5-a951-320fd5d4859e" path="/var/lib/kubelet/pods/7a0b10d1-eb58-44c5-a951-320fd5d4859e/volumes" Oct 30 00:55:57.964582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:57.964553 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:58.145674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:58.145641 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 00:55:58.176431 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:58.176397 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" event={"ID":"2f92dfc8-9e80-479a-b4de-05e0e183ea24","Type":"ContainerStarted","Data":"177f59b3eb3edddb19ba04cd39fd298a718b1f3f15b0153c3781758f28e0557b"} Oct 30 00:55:58.272887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:58.272847 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 00:55:58.273591 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:55:58.273565 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fae86f4_27c3_463e_9ec7_d3884a64c46f.slice/crio-e9de2f2b27ec165d68642970bfe8977a81de7f27753cae40d79c2d4aff5c8b58 WatchSource:0}: Error finding container e9de2f2b27ec165d68642970bfe8977a81de7f27753cae40d79c2d4aff5c8b58: Status 404 returned error can't find the container with id e9de2f2b27ec165d68642970bfe8977a81de7f27753cae40d79c2d4aff5c8b58 Oct 30 00:55:58.497963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:58.497816 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:58.835547 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:55:58.835472 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:55:59.185765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.185730 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" event={"ID":"2f92dfc8-9e80-479a-b4de-05e0e183ea24","Type":"ContainerStarted","Data":"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece"} Oct 30 00:55:59.185977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.185795 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:55:59.187473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.187449 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" event={"ID":"1fae86f4-27c3-463e-9ec7-d3884a64c46f","Type":"ContainerStarted","Data":"2f61f03ab94ab3fe973e46d151872a2ad25e1ad55e2553101c063f6ca7016eec"} Oct 30 00:55:59.187576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.187488 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" event={"ID":"1fae86f4-27c3-463e-9ec7-d3884a64c46f","Type":"ContainerStarted","Data":"e9de2f2b27ec165d68642970bfe8977a81de7f27753cae40d79c2d4aff5c8b58"} Oct 30 00:55:59.187576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.187531 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:55:59.227927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.227883 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" podStartSLOduration=2.227857697 podStartE2EDuration="2.227857697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.224065397 +0000 UTC m=+1411.464300512" watchObservedRunningTime="2026-10-30 00:55:59.227857697 +0000 UTC m=+1411.468092806" Oct 30 00:55:59.271316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:55:59.265777 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" podStartSLOduration=2.26576156 podStartE2EDuration="2.26576156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:59.262064251 +0000 UTC m=+1411.502299363" watchObservedRunningTime="2026-10-30 00:55:59.26576156 +0000 UTC m=+1411.505996675" Oct 30 00:56:00.305201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:00.305135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:01.916965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:01.916933 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:02.057902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:02.057846 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:03.467166 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:03.467135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.055953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:04.055909 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.295888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:04.289963 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.440272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:04.440238 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.791570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:04.791498 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:06.296218 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:06.296179 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:07.879188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:07.879159 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:09.577039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:09.576929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:10.197933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:10.197900 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 00:56:10.198202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:10.198182 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 00:56:11.033131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.033098 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt"] Oct 30 00:56:11.037131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.037108 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8535811f-f4jzt"] Oct 30 00:56:11.146795 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.146765 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.570895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.568807 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.708117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.708081 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:11.830782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.830705 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:56:11.831053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.831003 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" containerID="cri-o://70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb" gracePeriod=30 Oct 30 00:56:11.877466 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.877438 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50919b3-d60b-4b6e-94dc-7d54fd10cb1d" path="/var/lib/kubelet/pods/b50919b3-d60b-4b6e-94dc-7d54fd10cb1d/volumes" Oct 30 00:56:11.890604 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:11.890574 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.978048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:12.978017 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:13.517935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:13.515366 5938 patch_prober.go:29] interesting pod/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:56:13.517935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:13.515428 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:56:14.819065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:14.819030 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:16.648373 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:16.648341 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:17.161508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.161481 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:56:17.162358 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.162337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:17.256814 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.256726 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.256827 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.256911 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.256988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.257035 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.257138 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257285 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.257210 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.257304 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.257702 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.257342 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\") pod \"e369062e-6423-4537-91fa-55edba20c3af\" (UID: \"e369062e-6423-4537-91fa-55edba20c3af\") " Oct 30 00:56:17.258658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.258624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:17.260172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.260053 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:17.260172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.260136 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:17.260341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.260199 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:17.260501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.260478 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:17.261743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.261715 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:17.263157 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.263133 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:17.263963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.263934 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "kube-api-access-cb8sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:17.264047 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.264004 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo" pod "e369062e-6423-4537-91fa-55edba20c3af" (UID: "e369062e-6423-4537-91fa-55edba20c3af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:56:17.327671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.327591 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:17.360957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360911 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.360957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360935 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.360957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360946 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.360957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360956 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.360957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360965 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e369062e-6423-4537-91fa-55edba20c3af-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.361268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360973 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8sm\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-kube-api-access-cb8sm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.361268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360981 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e369062e-6423-4537-91fa-55edba20c3af-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.361268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360991 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e369062e-6423-4537-91fa-55edba20c3af-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.361268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.360999 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e369062e-6423-4537-91fa-55edba20c3af-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:17.362782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.362757 5938 generic.go:356] "Generic (PLEG): container finished" podID="e369062e-6423-4537-91fa-55edba20c3af" containerID="70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb" exitCode=0 Oct 30 00:56:17.362965 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.362944 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerDied","Data":"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb"} Oct 30 00:56:17.363066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.363020 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" event={"ID":"e369062e-6423-4537-91fa-55edba20c3af","Type":"ContainerDied","Data":"89a78edcd6e98e73c516338dd84a2d9aeea0569475d98f5c377133978da90435"} Oct 30 00:56:17.363136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.363070 5938 scope.go:122] "RemoveContainer" containerID="70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb" Oct 30 00:56:17.363315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.363267 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s" Oct 30 00:56:17.381213 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.381149 5938 scope.go:122] "RemoveContainer" containerID="921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f" Oct 30 00:56:17.423511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.423461 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:56:17.427295 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.427274 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d788ba-openshift-default-7f77dc57fc-s9h8s"] Oct 30 00:56:17.427547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.427534 5938 scope.go:122] "RemoveContainer" containerID="70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb" Oct 30 00:56:17.428061 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:17.427971 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb\": container with ID starting with 70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb not found: ID does not exist" containerID="70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb" Oct 30 00:56:17.428061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.427996 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb"} err="failed to get container status \"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb\": rpc error: code = NotFound desc = could not find container \"70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb\": container with ID starting with 70b66047e9ef872d423f18c23698abf94a1d87f91ba302b0a514cc96cd6254bb not found: ID does not exist" Oct 30 00:56:17.428061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.428013 5938 scope.go:122] "RemoveContainer" containerID="921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f" Oct 30 00:56:17.428511 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:17.428495 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f\": container with ID starting with 921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f not found: ID does not exist" containerID="921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f" Oct 30 00:56:17.428587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.428514 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f"} err="failed to get container status \"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f\": rpc error: code = NotFound desc = could not find container \"921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f\": container with ID starting with 921c96cdafcdfc622abfe63ee072f7d41733118a34c0f97385e1fd27f44ad43f not found: ID does not exist" Oct 30 00:56:17.878759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:17.878722 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e369062e-6423-4537-91fa-55edba20c3af" path="/var/lib/kubelet/pods/e369062e-6423-4537-91fa-55edba20c3af/volumes" Oct 30 00:56:18.228931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:18.228794 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:19.856083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:19.856044 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:20.510710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:20.510676 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:22.033648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:22.033609 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn"] Oct 30 00:56:22.044674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:22.044644 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df76b0f8-5hwsn"] Oct 30 00:56:22.406045 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:22.406012 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.005121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:23.005089 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.143167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:23.143135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.824154 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:23.824118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:23.877608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:23.877578 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325a495b-a3b8-45fb-8eec-dd6e66494b6f" path="/var/lib/kubelet/pods/325a495b-a3b8-45fb-8eec-dd6e66494b6f/volumes" Oct 30 00:56:24.081474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.081403 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 00:56:24.082250 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.082232 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.082312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.082252 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.082312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.082274 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.082312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.082280 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.082413 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.082402 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.087474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.087455 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.110611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.110586 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 00:56:24.118758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.118735 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b3bdf9-serving-cert\"" Oct 30 00:56:24.130663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.130633 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84lv\" (UniqueName: \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.130803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.130690 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.130803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.130763 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.130933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.130840 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.210561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.210530 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:24.232522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.232476 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84lv\" (UniqueName: \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.232657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.232539 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.232657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.232615 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.232738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.232692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.233150 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:24.233124 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-serving-cert: secret "maas-api-e2e-inf-b-b3bdf9-serving-cert" not found Oct 30 00:56:24.233274 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:24.233208 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls podName:2a785cc9-3c64-4ddf-91c8-0b75172f13d7 nodeName:}" failed. No retries permitted until 2026-10-30 00:56:24.733186095 +0000 UTC m=+1436.973421187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls") pod "maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" (UID: "2a785cc9-3c64-4ddf-91c8-0b75172f13d7") : secret "maas-api-e2e-inf-b-b3bdf9-serving-cert" not found Oct 30 00:56:24.234573 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.234543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.240138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.240113 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.241695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.241666 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 00:56:24.242950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.242922 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e369062e-6423-4537-91fa-55edba20c3af" containerName="istio-proxy" Oct 30 00:56:24.258624 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.258603 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.259380 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.259358 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84lv\" (UniqueName: \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.289057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.288285 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 00:56:24.289288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.289256 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bab19a-openshift-default-dockercfg-t7s7x\"" Oct 30 00:56:24.334210 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334144 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334280 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334334 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334433 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334531 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334515 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334581 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334646 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.334782 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334759 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8h6\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.335032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.334890 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.354845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.354820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.444964 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445102 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445298 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445342 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445412 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8h6\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.445884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445479 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.446540 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.445994 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.480487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.480456 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.499217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.499183 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.510896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.500285 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.511167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.506154 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.511336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.506222 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.511488 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.507000 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:24.518885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.507572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.518885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.517061 5938 generic.go:356] "Generic (PLEG): container finished" podID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerID="e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10" exitCode=0 Oct 30 00:56:24.518885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.517194 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerDied","Data":"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10"} Oct 30 00:56:24.518885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.517730 5938 scope.go:122] "RemoveContainer" containerID="e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10" Oct 30 00:56:24.535926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.520827 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:24.535926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.520958 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:24.535926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.521017 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:24.551937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.551907 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.563934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.555910 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8h6\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\") pod \"e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.641591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.641452 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:24.758933 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.758857 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.762259 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.762233 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:24.976534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.976503 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 00:56:24.977898 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:24.977834 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874ef415_0d90_4025_a6ec_089a8cd4933d.slice/crio-ed25be0cdafcbd72015b859949cb85448556ea560c496202ac5c227f7e27c37e WatchSource:0}: Error finding container ed25be0cdafcbd72015b859949cb85448556ea560c496202ac5c227f7e27c37e: Status 404 returned error can't find the container with id ed25be0cdafcbd72015b859949cb85448556ea560c496202ac5c227f7e27c37e Oct 30 00:56:24.980595 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.980556 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:24.980680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.980627 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:24.980680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:24.980655 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:25.005513 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.005493 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:25.330895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.330839 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 00:56:25.332506 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:25.332472 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a785cc9_3c64_4ddf_91c8_0b75172f13d7.slice/crio-4cdf765f4ef6a1286fb68e541a7994858b607971f960b9663ca564eabcc605ba WatchSource:0}: Error finding container 4cdf765f4ef6a1286fb68e541a7994858b607971f960b9663ca564eabcc605ba: Status 404 returned error can't find the container with id 4cdf765f4ef6a1286fb68e541a7994858b607971f960b9663ca564eabcc605ba Oct 30 00:56:25.530091 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.530053 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerStarted","Data":"613c85d3764687b299cea8049862bb9e561ef82d060b69c13a5773a112410193"} Oct 30 00:56:25.530298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.530156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerStarted","Data":"ed25be0cdafcbd72015b859949cb85448556ea560c496202ac5c227f7e27c37e"} Oct 30 00:56:25.532654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.532619 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerStarted","Data":"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6"} Oct 30 00:56:25.534649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.534624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" event={"ID":"2a785cc9-3c64-4ddf-91c8-0b75172f13d7","Type":"ContainerStarted","Data":"4cdf765f4ef6a1286fb68e541a7994858b607971f960b9663ca564eabcc605ba"} Oct 30 00:56:25.575381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.575334 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podStartSLOduration=1.57531681 podStartE2EDuration="1.57531681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:25.573190307 +0000 UTC m=+1437.813425449" watchObservedRunningTime="2026-10-30 00:56:25.57531681 +0000 UTC m=+1437.815551922" Oct 30 00:56:25.642535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:25.642502 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:26.252262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.252229 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:56:26.253553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.253526 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:26.253668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.253571 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:26.470222 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.470191 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:26.544234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.544134 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" event={"ID":"2a785cc9-3c64-4ddf-91c8-0b75172f13d7","Type":"ContainerStarted","Data":"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea"} Oct 30 00:56:26.544234 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.544210 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:26.611038 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.610976 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" podStartSLOduration=2.262941413 podStartE2EDuration="2.610959812s" totalImagesPullingTime="348.018399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:26.59131067 +0000 UTC m=+1438.831545784" watchObservedRunningTime="2026-10-30 00:56:26.610959812 +0000 UTC m=+1438.851194925" Oct 30 00:56:26.649398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:26.649370 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:27.252676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:27.252647 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:27.252885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:27.252698 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:27.551956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:27.551874 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:27.552714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:27.552694 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:56:28.022842 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.022777 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:28.211349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.211308 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:28.252424 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.252394 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:28.252575 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.252438 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:28.378104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.378048 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:28.836005 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:28.835850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:28.842467 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:28.842439 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:56:29.143381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.141216 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.252844 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.252814 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:29.253061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.252968 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:29.309925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.305351 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.577934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.577833 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.828280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.828204 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.967048 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.967018 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:56:29.971224 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.971202 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:29.971740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.971714 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.000022 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:29.999996 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bab19a-dockercfg-8bhgl\"" Oct 30 00:56:30.000884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.000845 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bab19a\"" Oct 30 00:56:30.011407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.011385 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:56:30.012201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.012116 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:56:30.199219 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.199075 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbkt\" (UniqueName: \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.243289 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.243256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.299884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.299344 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:30.299884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.299421 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:30.311884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.303883 5938 scope.go:122] "RemoveContainer" containerID="79028b273516a60422cd13f15a3d4c2ece3c50231ab7ea98781d0b489c93db39" Oct 30 00:56:30.345388 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.345352 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbkt\" (UniqueName: \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.345537 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.345437 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.346561 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.346536 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.347962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.347934 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:56:30.353793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.352597 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:56:30.353793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.352680 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.357504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.357432 5938 scope.go:122] "RemoveContainer" containerID="3ed6cc0bdb18622edb40369b10bab1e390bd9fdc19d26d53771fc56cdf2103c0" Oct 30 00:56:30.369000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.368931 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbkt\" (UniqueName: \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\") pod \"payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.369914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.369896 5938 scope.go:122] "RemoveContainer" containerID="21e23dce8ad8cf74929aba9b9bbd6c2e6bf3c5c90a6d7dcd5aa9c64bf4b70430" Oct 30 00:56:30.447294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.447261 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.447544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.447520 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2b8\" (UniqueName: \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.549069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.549038 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.549257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.549243 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2b8\" (UniqueName: \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.549686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.549668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.571887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.571518 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2b8\" (UniqueName: \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\") pod \"payload-processing-e2e-api-b-bab19a-665c67756-zz5k5\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.595892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.588045 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:30.667882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.666614 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:30.926178 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:30.926143 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:56:30.928472 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:30.928442 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd378156c_7b5b_4b2c_b2bb_0daedbc3908b.slice/crio-f143192fdc47959939a95e8bde3de3a1f2d25336e47950f649609a0834844919 WatchSource:0}: Error finding container f143192fdc47959939a95e8bde3de3a1f2d25336e47950f649609a0834844919: Status 404 returned error can't find the container with id f143192fdc47959939a95e8bde3de3a1f2d25336e47950f649609a0834844919 Oct 30 00:56:31.016025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.015997 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:56:31.017951 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:31.017923 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6eb19f7_f78e_41f7_b1c4_4b3948116b29.slice/crio-8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba WatchSource:0}: Error finding container 8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba: Status 404 returned error can't find the container with id 8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba Oct 30 00:56:31.226233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.226126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:31.253453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.253420 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 00:56:31.253652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.253476 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 00:56:31.651066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.651030 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" event={"ID":"f6eb19f7-f78e-41f7-b1c4-4b3948116b29","Type":"ContainerStarted","Data":"ca03814a44410ef772c7df306ed491b2804a5f0e7496681dfe02336224066bda"} Oct 30 00:56:31.651257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.651096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" event={"ID":"f6eb19f7-f78e-41f7-b1c4-4b3948116b29","Type":"ContainerStarted","Data":"8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba"} Oct 30 00:56:31.651257 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.651126 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:31.652826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.652800 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" event={"ID":"d378156c-7b5b-4b2c-b2bb-0daedbc3908b","Type":"ContainerStarted","Data":"a93f845aa0a4eeb37b29575c41e9362d84ad9567c99957200d967527682a3a39"} Oct 30 00:56:31.653003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.652842 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" event={"ID":"d378156c-7b5b-4b2c-b2bb-0daedbc3908b","Type":"ContainerStarted","Data":"f143192fdc47959939a95e8bde3de3a1f2d25336e47950f649609a0834844919"} Oct 30 00:56:31.653003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.652894 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:31.719487 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.719446 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" podStartSLOduration=1.719431105 podStartE2EDuration="1.719431105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:31.716102589 +0000 UTC m=+1443.956337712" watchObservedRunningTime="2026-10-30 00:56:31.719431105 +0000 UTC m=+1443.959666219" Oct 30 00:56:31.772981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:31.772937 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" podStartSLOduration=2.772924733 podStartE2EDuration="2.772924733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:31.771527881 +0000 UTC m=+1444.011762994" watchObservedRunningTime="2026-10-30 00:56:31.772924733 +0000 UTC m=+1444.013159846" Oct 30 00:56:32.256409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:32.256382 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:56:32.257482 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:32.257463 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:56:32.557700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:32.557627 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 00:56:33.170130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:33.170097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:34.984743 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:34.984706 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.777076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:35.777044 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.915112 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:35.915073 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.127152 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.127120 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.282765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.282373 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.609648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.609608 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 00:56:36.613228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.613208 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.653287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.653263 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-481ab7-openshift-default-dockercfg-lttfj\"" Oct 30 00:56:36.657978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.657952 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 00:56:36.701533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.701499 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773367 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773452 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773658 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jh7f\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773704 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773788 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.773852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.774016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.774248 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.774901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.774296 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.875754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.875657 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jh7f\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876019 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876107 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876168 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876292 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876204 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876346 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876320 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876359 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876419 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876470 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876486 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.876520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876499 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.877016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.876992 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.877153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.877132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.877199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.877151 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.879701 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.879671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.880255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.880232 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.900610 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.900586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jh7f\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.901171 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.901150 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\") pod \"e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:36.931619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:36.931592 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:37.008919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.006301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:37.262444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.262408 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 00:56:37.263035 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:37.263011 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa93d7f0_cff8_41be_9296_ac152e0cd370.slice/crio-9c47aef421ff2b0060f3df2cdcb4661985908d1ab76a5b27cf0f20e99c9e87b0 WatchSource:0}: Error finding container 9c47aef421ff2b0060f3df2cdcb4661985908d1ab76a5b27cf0f20e99c9e87b0: Status 404 returned error can't find the container with id 9c47aef421ff2b0060f3df2cdcb4661985908d1ab76a5b27cf0f20e99c9e87b0 Oct 30 00:56:37.265703 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.265667 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:37.265813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.265735 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:37.265813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.265768 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:37.318083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.312698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:37.471907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.471833 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:37.736906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.734604 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerStarted","Data":"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100"} Oct 30 00:56:37.736906 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.734687 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerStarted","Data":"9c47aef421ff2b0060f3df2cdcb4661985908d1ab76a5b27cf0f20e99c9e87b0"} Oct 30 00:56:37.797787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.797757 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:37.812938 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.812853 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podStartSLOduration=1.8128401969999999 podStartE2EDuration="1.812840197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:37.801037003 +0000 UTC m=+1450.041272118" watchObservedRunningTime="2026-10-30 00:56:37.812840197 +0000 UTC m=+1450.053075310" Oct 30 00:56:37.932139 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:37.932116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:38.244884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.244840 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 00:56:38.248880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.248842 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.267712 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.267682 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8dbe73-openshift-default-dockercfg-hjfpm\"" Oct 30 00:56:38.284705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.284679 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 00:56:38.345891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.345852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.433621 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.433803 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.433883 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.433942 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.434016 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.434129 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.434167 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgl9\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.434200 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.436953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.434239 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.494138 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.494103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:38.535436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535360 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535403 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgl9\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535448 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535548 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535672 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535719 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535760 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.535943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.535813 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.536123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.536067 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.536239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.536212 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.536319 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.536242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.536319 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.536264 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.536600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.536580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.538205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.538182 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.538420 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.538403 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.557716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.557695 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.558083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.558061 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgl9\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\") pod \"e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.565017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.564998 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:38.886408 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.886380 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 00:56:38.887468 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:38.887440 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b943865_a7ab_462c_b1ce_3f85e0146fdf.slice/crio-8d363718cfc6a5ea1d9b96126ec9622f9f881cca096ecaabb91cf95088e87d11 WatchSource:0}: Error finding container 8d363718cfc6a5ea1d9b96126ec9622f9f881cca096ecaabb91cf95088e87d11: Status 404 returned error can't find the container with id 8d363718cfc6a5ea1d9b96126ec9622f9f881cca096ecaabb91cf95088e87d11 Oct 30 00:56:38.890284 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.890250 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:38.890371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.890312 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:38.890371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.890343 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:56:38.904203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.904152 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:38.943519 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:38.943494 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:39.397959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.397922 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:39.753957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.753828 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerStarted","Data":"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596"} Oct 30 00:56:39.754336 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.754309 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerStarted","Data":"8d363718cfc6a5ea1d9b96126ec9622f9f881cca096ecaabb91cf95088e87d11"} Oct 30 00:56:39.754576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.754552 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:39.758730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.758706 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:56:39.823083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:39.823033 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podStartSLOduration=1.82301647 podStartE2EDuration="1.82301647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:39.808445675 +0000 UTC m=+1452.048680789" watchObservedRunningTime="2026-10-30 00:56:39.82301647 +0000 UTC m=+1452.063251583" Oct 30 00:56:40.423040 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.423008 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:40.565266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.565237 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:40.571021 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.570997 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:40.620344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.620316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:40.762748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.762674 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:40.763440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:40.763419 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:56:41.171759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:41.171723 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:41.757276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:41.755427 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:42.182793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.182758 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:42.265397 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.265369 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 00:56:42.268836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.268814 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.285172 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.285128 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-481ab7\"" Oct 30 00:56:42.289902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.289848 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 00:56:42.340618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.340581 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 00:56:42.344923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.344903 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.353286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.353262 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:42.359590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.359564 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-481ab7-dockercfg-gpk2z\"" Oct 30 00:56:42.362840 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.362815 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 00:56:42.418660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.418632 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdx7d\" (UniqueName: \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.418903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.418851 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.495633 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.495560 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:42.519973 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.519948 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhl65\" (UniqueName: \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.520096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.519984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.520254 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.520232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.520303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.520282 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdx7d\" (UniqueName: \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.520778 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.520760 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.591883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.584744 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdx7d\" (UniqueName: \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\") pod \"payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.642940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.639930 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhl65\" (UniqueName: \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.642940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.640115 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.642940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.641123 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.695011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.691102 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhl65\" (UniqueName: \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\") pod \"payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:42.746943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.745962 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:56:42.825299 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.825177 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:56:42.883760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.883715 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:42.959371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:42.959273 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:43.209831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.209800 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 00:56:43.210009 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:56:43.209989 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0e83c1_6a4a_4a69_81a8_e352193bba3b.slice/crio-91cec4bd0c4681f51908707b2830244cf014fcee1b7a6fff25712c34b89d97f5 WatchSource:0}: Error finding container 91cec4bd0c4681f51908707b2830244cf014fcee1b7a6fff25712c34b89d97f5: Status 404 returned error can't find the container with id 91cec4bd0c4681f51908707b2830244cf014fcee1b7a6fff25712c34b89d97f5 Oct 30 00:56:43.307313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.307278 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 00:56:43.842121 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.842086 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" event={"ID":"3f0e83c1-6a4a-4a69-81a8-e352193bba3b","Type":"ContainerStarted","Data":"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf"} Oct 30 00:56:43.842601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.842153 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" event={"ID":"3f0e83c1-6a4a-4a69-81a8-e352193bba3b","Type":"ContainerStarted","Data":"91cec4bd0c4681f51908707b2830244cf014fcee1b7a6fff25712c34b89d97f5"} Oct 30 00:56:43.842601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.842181 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:43.844054 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.844019 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" event={"ID":"728bfdfc-545c-4608-b7ca-cd42133ac9cf","Type":"ContainerStarted","Data":"ae3367542c00678e6fee8ae8319b0d822712c3535e32e21c3497a4d91460664f"} Oct 30 00:56:43.844227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.844074 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" event={"ID":"728bfdfc-545c-4608-b7ca-cd42133ac9cf","Type":"ContainerStarted","Data":"b6c37bcb5dff958132ba830eb29d5a3da5fe6fe2af627266e5918798c06310af"} Oct 30 00:56:43.844227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.844118 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:43.889715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.889665 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" podStartSLOduration=1.8896505399999999 podStartE2EDuration="1.88965054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:43.88616918 +0000 UTC m=+1456.126404294" watchObservedRunningTime="2026-10-30 00:56:43.88965054 +0000 UTC m=+1456.129885653" Oct 30 00:56:43.924903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.924792 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" podStartSLOduration=1.924774381 podStartE2EDuration="1.924774381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:43.924487544 +0000 UTC m=+1456.164722658" watchObservedRunningTime="2026-10-30 00:56:43.924774381 +0000 UTC m=+1456.165009495" Oct 30 00:56:43.959794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:43.959766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:44.133691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:44.133656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:44.297801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:44.297760 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:45.396804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:45.396774 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:56:45.428383 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:45.428351 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:56:45.568077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:45.568043 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:45.723583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:45.723504 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:46.198559 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:46.198521 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:47.542761 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:47.542720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:47.684130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:47.684093 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:47.866606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:47.866571 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:56:47.866851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:47.866822 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" containerID="cri-o://eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17" gracePeriod=30 Oct 30 00:56:47.943501 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:47.943472 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:48.472339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:48.472294 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:50.403834 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.403790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:50.429502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.429464 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" podUID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" containerName="payload-pre-processing" containerID="cri-o://935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9" gracePeriod=25 Oct 30 00:56:50.726911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.726888 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:56:50.813801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.813766 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\" (UniqueName: \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\") pod \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " Oct 30 00:56:50.813999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.813812 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\") pod \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\" (UID: \"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c\") " Oct 30 00:56:50.814349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.814314 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume" pod "7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" (UID: "7fa1136a-ea8d-42ec-bbcd-c777768d6c6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:50.814836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.814809 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:50.816206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.816174 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p" pod "7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" (UID: "7fa1136a-ea8d-42ec-bbcd-c777768d6c6c"). InnerVolumeSpecName "kube-api-access-ppd2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:50.859194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.859161 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" podUID="181fdbda-e2e0-4639-84c1-fe7d2113131d" containerName="payload-processing" containerID="cri-o://9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652" gracePeriod=25 Oct 30 00:56:50.905812 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.905790 5938 generic.go:356] "Generic (PLEG): container finished" podID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" containerID="935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9" exitCode=0 Oct 30 00:56:50.905943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.905831 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" Oct 30 00:56:50.905943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.905891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" event={"ID":"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c","Type":"ContainerDied","Data":"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9"} Oct 30 00:56:50.905943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.905928 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d" event={"ID":"7fa1136a-ea8d-42ec-bbcd-c777768d6c6c","Type":"ContainerDied","Data":"4fea11969b104412eb34d213a531a95fac0206e7674b8f15a59e1e75260c266b"} Oct 30 00:56:50.906083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.905949 5938 scope.go:122] "RemoveContainer" containerID="935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9" Oct 30 00:56:50.916366 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.916341 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppd2p\" (UniqueName: \"kubernetes.io/projected/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c-kube-api-access-ppd2p\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:50.970084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.970025 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:56:50.973670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.973651 5938 scope.go:122] "RemoveContainer" containerID="935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9" Oct 30 00:56:50.974808 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:50.974789 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9\": container with ID starting with 935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9 not found: ID does not exist" containerID="935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9" Oct 30 00:56:50.974978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.974816 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9"} err="failed to get container status \"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9\": rpc error: code = NotFound desc = could not find container \"935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9\": container with ID starting with 935f8578885e6ff5f81c426cb686f46f70c155a4d1921317a756e7bde51305c9 not found: ID does not exist" Oct 30 00:56:50.975967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:50.975949 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52785f5e-67559cc487-mbl9d"] Oct 30 00:56:51.120926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.120900 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:56:51.219796 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.219765 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\" (UniqueName: \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\") pod \"181fdbda-e2e0-4639-84c1-fe7d2113131d\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " Oct 30 00:56:51.219978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.219896 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\") pod \"181fdbda-e2e0-4639-84c1-fe7d2113131d\" (UID: \"181fdbda-e2e0-4639-84c1-fe7d2113131d\") " Oct 30 00:56:51.220305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.220253 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume" pod "181fdbda-e2e0-4639-84c1-fe7d2113131d" (UID: "181fdbda-e2e0-4639-84c1-fe7d2113131d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:51.222104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.222085 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554" pod "181fdbda-e2e0-4639-84c1-fe7d2113131d" (UID: "181fdbda-e2e0-4639-84c1-fe7d2113131d"). InnerVolumeSpecName "kube-api-access-bv554". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:51.321541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.321509 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv554\" (UniqueName: \"kubernetes.io/projected/181fdbda-e2e0-4639-84c1-fe7d2113131d-kube-api-access-bv554\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:51.321541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.321533 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181fdbda-e2e0-4639-84c1-fe7d2113131d-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:51.879871 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.879824 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" path="/var/lib/kubelet/pods/7fa1136a-ea8d-42ec-bbcd-c777768d6c6c/volumes" Oct 30 00:56:51.914831 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.914803 5938 generic.go:356] "Generic (PLEG): container finished" podID="181fdbda-e2e0-4639-84c1-fe7d2113131d" containerID="9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652" exitCode=0 Oct 30 00:56:51.914987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.914846 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" Oct 30 00:56:51.914987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.914896 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" event={"ID":"181fdbda-e2e0-4639-84c1-fe7d2113131d","Type":"ContainerDied","Data":"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652"} Oct 30 00:56:51.914987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.914950 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5" event={"ID":"181fdbda-e2e0-4639-84c1-fe7d2113131d","Type":"ContainerDied","Data":"d498c9ee62caea96b7a36adafff24a147346683d40044e4ab2d0832dbc5e5449"} Oct 30 00:56:51.914987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.914972 5938 scope.go:122] "RemoveContainer" containerID="9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652" Oct 30 00:56:51.925353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.925333 5938 scope.go:122] "RemoveContainer" containerID="9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652" Oct 30 00:56:51.925613 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:51.925595 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652\": container with ID starting with 9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652 not found: ID does not exist" containerID="9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652" Oct 30 00:56:51.925662 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.925620 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652"} err="failed to get container status \"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652\": rpc error: code = NotFound desc = could not find container \"9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652\": container with ID starting with 9068e55ca4ca422355a53f151569c96ecb8df62a6926d1a14810424fc776b652 not found: ID does not exist" Oct 30 00:56:51.969803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.969782 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:56:51.980876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:51.980843 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-52785f5e-5f57cff566-5jzd5"] Oct 30 00:56:52.118432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:52.118409 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:53.150970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.150909 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:56:53.256288 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256251 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.256584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256337 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.256584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256454 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.256584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256501 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.256584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256535 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.256584 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256578 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.257161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256676 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.257161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256910 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.257161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256996 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\") pod \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\" (UID: \"8278bb98-db1b-480a-a6f3-c96dc06ee69d\") " Oct 30 00:56:53.257161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.256531 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:53.257161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.257042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:53.257732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.257702 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.257732 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.257731 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.258298 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.258271 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:53.258534 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.258510 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:53.258858 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.258833 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:56:53.259395 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.259371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:56:53.259588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.259551 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:56:53.261305 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.261276 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "kube-api-access-8dpvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:53.263026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.263002 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token" pod "8278bb98-db1b-480a-a6f3-c96dc06ee69d" (UID: "8278bb98-db1b-480a-a6f3-c96dc06ee69d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:56:53.359325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359302 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359328 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359339 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359347 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359355 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359365 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpvm\" (UniqueName: \"kubernetes.io/projected/8278bb98-db1b-480a-a6f3-c96dc06ee69d-kube-api-access-8dpvm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.359479 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.359375 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8278bb98-db1b-480a-a6f3-c96dc06ee69d-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:56:53.881971 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.881932 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181fdbda-e2e0-4639-84c1-fe7d2113131d" path="/var/lib/kubelet/pods/181fdbda-e2e0-4639-84c1-fe7d2113131d/volumes" Oct 30 00:56:53.931449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.931418 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:53.934922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.934845 5938 generic.go:356] "Generic (PLEG): container finished" podID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerID="eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17" exitCode=0 Oct 30 00:56:53.935170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.935090 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" Oct 30 00:56:53.949565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.935466 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerDied","Data":"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17"} Oct 30 00:56:53.949565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.935528 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg" event={"ID":"8278bb98-db1b-480a-a6f3-c96dc06ee69d","Type":"ContainerDied","Data":"ed260754d17c7d17f24a29eb5b50d84807a9e1a58478323f3e559fb7ca01d51a"} Oct 30 00:56:53.949565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.935560 5938 scope.go:122] "RemoveContainer" containerID="eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17" Oct 30 00:56:53.966447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.966420 5938 scope.go:122] "RemoveContainer" containerID="f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a" Oct 30 00:56:53.977304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.977281 5938 scope.go:122] "RemoveContainer" containerID="eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17" Oct 30 00:56:53.977623 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:53.977602 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17\": container with ID starting with eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17 not found: ID does not exist" containerID="eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17" Oct 30 00:56:53.977680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.977632 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17"} err="failed to get container status \"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17\": rpc error: code = NotFound desc = could not find container \"eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17\": container with ID starting with eee7863466b1d2f78692970989d98adf9bbeb34a7c5af0148325266b305abb17 not found: ID does not exist" Oct 30 00:56:53.977680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.977651 5938 scope.go:122] "RemoveContainer" containerID="f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a" Oct 30 00:56:53.977975 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:53.977953 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a\": container with ID starting with f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a not found: ID does not exist" containerID="f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a" Oct 30 00:56:53.978136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:53.977983 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a"} err="failed to get container status \"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a\": rpc error: code = NotFound desc = could not find container \"f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a\": container with ID starting with f32d78bf01c644e6942f29330b99bc741f9fb110a03034288a63b4ae0f37866a not found: ID does not exist" Oct 30 00:56:54.006032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:54.006003 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:56:54.017895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:54.017831 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-hbqrg"] Oct 30 00:56:54.854956 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:54.854926 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 00:56:54.855377 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:54.855005 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 00:56:55.633108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:55.633050 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:55.880699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:55.880664 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" path="/var/lib/kubelet/pods/8278bb98-db1b-480a-a6f3-c96dc06ee69d/volumes" Oct 30 00:56:57.467312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:57.467275 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:58.840048 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:56:58.840008 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:56:59.523880 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:56:59.523786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:00.407998 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.407959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:00.814014 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.813923 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 00:57:00.815447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.815416 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.815627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.815610 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.815734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.815722 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.815838 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.815824 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.816053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816025 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" containerName="payload-pre-processing" Oct 30 00:57:00.816053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816048 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" containerName="payload-pre-processing" Oct 30 00:57:00.816228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816097 5938 cpu_manager.go:414] "containerMap: removing container" podUID="181fdbda-e2e0-4639-84c1-fe7d2113131d" containerName="payload-processing" Oct 30 00:57:00.816228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816106 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181fdbda-e2e0-4639-84c1-fe7d2113131d" containerName="payload-processing" Oct 30 00:57:00.816338 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816318 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa1136a-ea8d-42ec-bbcd-c777768d6c6c" containerName="payload-pre-processing" Oct 30 00:57:00.816409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816348 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="181fdbda-e2e0-4639-84c1-fe7d2113131d" containerName="payload-processing" Oct 30 00:57:00.816409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816370 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.816409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.816385 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8278bb98-db1b-480a-a6f3-c96dc06ee69d" containerName="istio-proxy" Oct 30 00:57:00.820492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.820472 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:00.833801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.833773 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 00:57:00.841088 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.840332 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-48df96-openshift-default-dockercfg-h6qzs\"" Oct 30 00:57:00.909850 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:00.909813 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026279 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026336 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpxz\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026459 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026492 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026583 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026726 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.029893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.026773 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.053901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.052835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:01.129943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.129845 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.130610 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.131434 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.131106 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.131534 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.131810 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.132010 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpxz\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.133894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.132469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.152061 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.152019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.152506 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.152451 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.152745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.152730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.152978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.152962 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.153107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.153093 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.153277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.153179 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.156890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.153688 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.178907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.158307 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.178907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.175768 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.179262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.179242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpxz\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.438379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.438300 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:01.579469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.579427 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:01.728625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.728517 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:01.816260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.809085 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 00:57:01.818946 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:01.818911 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7f95f3_705b_480c_8c51_f9f24766b9f8.slice/crio-38e7117a7c139df03f5cbe7673b9ea98ea1e1affb2acd878c2e2cd7193f8dad2 WatchSource:0}: Error finding container 38e7117a7c139df03f5cbe7673b9ea98ea1e1affb2acd878c2e2cd7193f8dad2: Status 404 returned error can't find the container with id 38e7117a7c139df03f5cbe7673b9ea98ea1e1affb2acd878c2e2cd7193f8dad2 Oct 30 00:57:01.821787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.821751 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:01.821996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.821835 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:01.821996 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:01.821897 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:02.114931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.112529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerStarted","Data":"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83"} Oct 30 00:57:02.114931 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.112597 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerStarted","Data":"38e7117a7c139df03f5cbe7673b9ea98ea1e1affb2acd878c2e2cd7193f8dad2"} Oct 30 00:57:02.160082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.160024 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podStartSLOduration=2.160005532 podStartE2EDuration="2.160005532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:02.156457928 +0000 UTC m=+1474.396693044" watchObservedRunningTime="2026-10-30 00:57:02.160005532 +0000 UTC m=+1474.400240646" Oct 30 00:57:02.229591 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.229560 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:02.438676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.438585 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:02.444464 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:02.444440 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:03.126754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:03.126722 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:03.127905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:03.127839 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:57:03.828843 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:03.828811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:04.993877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:04.993818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:05.145601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:05.145542 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:05.805339 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:05.805285 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.302924 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.302882 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.701287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.701216 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.854209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.854175 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.858967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.858942 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 00:57:06.865723 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.865701 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:06.881755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.881729 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 00:57:06.888451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.888417 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-48df96-dockercfg-dcqdj\"" Oct 30 00:57:06.888451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.888428 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-48df96\"" Oct 30 00:57:06.904360 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.904333 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:06.904974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.904947 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpv8\" (UniqueName: \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:06.923214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.923187 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 00:57:06.978622 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:06.977520 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.035886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.009806 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 00:57:07.047969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.021422 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpv8\" (UniqueName: \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:07.119568 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.119532 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:07.120629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.120588 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:07.121060 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.121040 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpv8\" (UniqueName: \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\") pod \"payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:07.188206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.188136 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:07.221618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.221577 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbdr\" (UniqueName: \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.221763 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.221638 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.324644 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.324607 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbdr\" (UniqueName: \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.325016 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.324672 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.325982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.325943 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.348970 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.348934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbdr\" (UniqueName: \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\") pod \"payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.500203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.500171 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:07.562318 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.562291 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 00:57:07.563574 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:07.563514 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bb250b_c85b_4e1d_88bf_d001d593c29c.slice/crio-0462fc2e5d7fc07b8cf81a6b82ee68061afc3d0405a9eef57c15fba052fff65d WatchSource:0}: Error finding container 0462fc2e5d7fc07b8cf81a6b82ee68061afc3d0405a9eef57c15fba052fff65d: Status 404 returned error can't find the container with id 0462fc2e5d7fc07b8cf81a6b82ee68061afc3d0405a9eef57c15fba052fff65d Oct 30 00:57:07.817502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.817457 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 00:57:07.832066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:07.832036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.841828 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:07.841800 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83166205_d53c_4d66_8e4f_eb1ec8d92cbb.slice/crio-3479aaa37c98e80603fee26e8ea963f6a68088ba9786efa9cb147074c98b7b60 WatchSource:0}: Error finding container 3479aaa37c98e80603fee26e8ea963f6a68088ba9786efa9cb147074c98b7b60: Status 404 returned error can't find the container with id 3479aaa37c98e80603fee26e8ea963f6a68088ba9786efa9cb147074c98b7b60 Oct 30 00:57:08.214303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.214257 5938 generic.go:356] "Generic (PLEG): container finished" podID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerID="54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753" exitCode=0 Oct 30 00:57:08.214544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.214520 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerDied","Data":"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753"} Oct 30 00:57:08.215129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.215102 5938 scope.go:122] "RemoveContainer" containerID="54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753" Oct 30 00:57:08.217601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.217571 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:08.217721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.217644 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:08.217721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.217686 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:08.218458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.218040 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" event={"ID":"83166205-d53c-4d66-8e4f-eb1ec8d92cbb","Type":"ContainerStarted","Data":"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b"} Oct 30 00:57:08.218458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.218086 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" event={"ID":"83166205-d53c-4d66-8e4f-eb1ec8d92cbb","Type":"ContainerStarted","Data":"3479aaa37c98e80603fee26e8ea963f6a68088ba9786efa9cb147074c98b7b60"} Oct 30 00:57:08.218458 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.218167 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:08.222440 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.222414 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" event={"ID":"e3bb250b-c85b-4e1d-88bf-d001d593c29c","Type":"ContainerStarted","Data":"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e"} Oct 30 00:57:08.222613 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.222466 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" event={"ID":"e3bb250b-c85b-4e1d-88bf-d001d593c29c","Type":"ContainerStarted","Data":"0462fc2e5d7fc07b8cf81a6b82ee68061afc3d0405a9eef57c15fba052fff65d"} Oct 30 00:57:08.223180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.223157 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:08.410680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.410597 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" podStartSLOduration=2.4105763270000002 podStartE2EDuration="2.410576327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:08.341309124 +0000 UTC m=+1480.581544239" watchObservedRunningTime="2026-10-30 00:57:08.410576327 +0000 UTC m=+1480.650811441" Oct 30 00:57:08.412722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.412666 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" podStartSLOduration=2.412652741 podStartE2EDuration="2.412652741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:08.390360118 +0000 UTC m=+1480.630595243" watchObservedRunningTime="2026-10-30 00:57:08.412652741 +0000 UTC m=+1480.652887887" Oct 30 00:57:08.592771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.592732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:08.735486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.735403 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:08.875835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:08.874648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:09.143625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.143584 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:57:09.232198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.232159 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerStarted","Data":"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53"} Oct 30 00:57:09.272922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.265063 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:09.467301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.467218 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:57:09.468768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.468741 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 00:57:09.468923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:09.468790 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 00:57:10.468000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:10.467968 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 00:57:10.468437 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:10.468025 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 00:57:10.822182 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:10.822097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:11.467449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:11.467411 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 00:57:11.467670 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:11.467473 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 00:57:12.467329 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:12.467292 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 00:57:12.467758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:12.467357 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 00:57:12.808937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:12.808763 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:13.014756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:13.014729 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:13.158774 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:13.158722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:13.467710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:13.467634 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 00:57:13.467710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:13.467689 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 00:57:14.539784 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:14.539754 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:57:14.540402 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:14.540379 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:57:14.541674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:14.541652 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 00:57:14.854739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:14.854686 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:15.373194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:15.373166 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:15.528759 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:15.528730 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:15.604915 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:15.604881 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:57:15.605381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:15.605127 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" containerID="cri-o://bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc" gracePeriod=30 Oct 30 00:57:15.678399 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:15.678333 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:16.527164 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:16.527112 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.031648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:17.031576 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.196891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:17.191429 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.650180 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:17.650144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:19.125611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:19.125576 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:20.052097 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:20.052060 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:20.243962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:20.243928 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 00:57:20.253341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:20.253314 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 00:57:20.812928 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:20.812767 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.160096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.154770 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:57:21.362920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.362488 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.364678 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.364732 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.364857 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.364934 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.364973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.365011 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.365120 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.365174 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.366653 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.365210 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\") pod \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\" (UID: \"b32c0cbe-f198-4f79-9fe2-4c3eab951e19\") " Oct 30 00:57:21.376201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.375801 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:21.377727 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.376738 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:21.379503 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.379476 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "kube-api-access-jkmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:21.396879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.379948 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:21.396879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.380477 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:21.406494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.397752 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmdm\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-kube-api-access-jkmdm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.406494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.397782 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.406494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.397798 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.406494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.397812 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.406494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.397826 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.415161 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.415118 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:21.418160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.415223 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:21.418160 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.416320 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:21.434215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.421814 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token" pod "b32c0cbe-f198-4f79-9fe2-4c3eab951e19" (UID: "b32c0cbe-f198-4f79-9fe2-4c3eab951e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:21.474959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.474463 5938 generic.go:356] "Generic (PLEG): container finished" podID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerID="bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc" exitCode=0 Oct 30 00:57:21.474959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.474641 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerDied","Data":"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc"} Oct 30 00:57:21.474959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.474691 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" event={"ID":"b32c0cbe-f198-4f79-9fe2-4c3eab951e19","Type":"ContainerDied","Data":"8f2ca84b86d3017e139eafaa97f6f213d3fbcdc57746ad95253f9f57113cbf75"} Oct 30 00:57:21.474959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.474726 5938 scope.go:122] "RemoveContainer" containerID="bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc" Oct 30 00:57:21.474959 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.474951 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm" Oct 30 00:57:21.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.497904 5938 scope.go:122] "RemoveContainer" containerID="354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f" Oct 30 00:57:21.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.500347 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.500372 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.500385 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.500403 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b32c0cbe-f198-4f79-9fe2-4c3eab951e19-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:21.517889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.511852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.537960 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.537934 5938 scope.go:122] "RemoveContainer" containerID="bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc" Oct 30 00:57:21.538414 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:21.538378 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc\": container with ID starting with bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc not found: ID does not exist" containerID="bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc" Oct 30 00:57:21.538515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.538418 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc"} err="failed to get container status \"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc\": rpc error: code = NotFound desc = could not find container \"bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc\": container with ID starting with bdaad64537bb39be557f6c0faef1a3d4885c597947982e94ae7bfd0cf507b6cc not found: ID does not exist" Oct 30 00:57:21.538515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.538436 5938 scope.go:122] "RemoveContainer" containerID="354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f" Oct 30 00:57:21.538973 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:21.538937 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f\": container with ID starting with 354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f not found: ID does not exist" containerID="354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f" Oct 30 00:57:21.539079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.538978 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f"} err="failed to get container status \"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f\": rpc error: code = NotFound desc = could not find container \"354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f\": container with ID starting with 354746d8ff64a5de5fc73f5c60ab8d40ab636735b41ade827282f2c0c95b846f not found: ID does not exist" Oct 30 00:57:21.552382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.552355 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:57:21.556223 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.556200 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-52785f5e-openshift-default-69c968b885-7cdfm"] Oct 30 00:57:21.729255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.729215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.880489 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.880450 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" path="/var/lib/kubelet/pods/b32c0cbe-f198-4f79-9fe2-4c3eab951e19/volumes" Oct 30 00:57:21.928461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:21.928386 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:22.053055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.053019 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:57:22.053294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.053258 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" containerID="cri-o://9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8" gracePeriod=30 Oct 30 00:57:22.058000 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.057976 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:57:22.058198 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.058176 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" containerID="cri-o://540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08" gracePeriod=30 Oct 30 00:57:22.063968 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.063945 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:57:22.064953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.064925 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" containerID="cri-o://bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e" gracePeriod=30 Oct 30 00:57:22.079393 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.079370 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:57:22.079597 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.079575 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" containerID="cri-o://64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98" gracePeriod=30 Oct 30 00:57:22.384027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:22.383993 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.129343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.129309 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.220953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.220921 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 00:57:23.221811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.221787 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.221811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.221810 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.222057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.222036 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.222113 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.222059 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.231716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.231690 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 00:57:23.233039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.233015 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.233199 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.233184 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32c0cbe-f198-4f79-9fe2-4c3eab951e19" containerName="istio-proxy" Oct 30 00:57:23.235673 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.235647 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.241095 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.241058 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 00:57:23.241304 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.241290 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 00:57:23.241461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.241169 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.250421 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.250405 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51b95e\"" Oct 30 00:57:23.262492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.262473 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51b95e-dockercfg-kh7st\"" Oct 30 00:57:23.268512 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.268484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.365886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.360997 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsplb\" (UniqueName: \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.365886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.361176 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvg5d\" (UniqueName: \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.365886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.361217 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.365886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.361289 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.471956 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.472203 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsplb\" (UniqueName: \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.472368 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvg5d\" (UniqueName: \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.472419 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.473388 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.475954 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.474172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.515887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.513601 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsplb\" (UniqueName: \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\") pod \"payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.548033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.541272 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvg5d\" (UniqueName: \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\") pod \"payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.571937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.567140 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:23.571937 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.568266 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:23.967695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.967668 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 00:57:23.968262 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:23.968235 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbed9a1_0ed5_4042_8979_691eb3cb6f0b.slice/crio-6bfc484692dc25289fd35fa27c2f30c3369cd5f09256dca07628a1ceb5362076 WatchSource:0}: Error finding container 6bfc484692dc25289fd35fa27c2f30c3369cd5f09256dca07628a1ceb5362076: Status 404 returned error can't find the container with id 6bfc484692dc25289fd35fa27c2f30c3369cd5f09256dca07628a1ceb5362076 Oct 30 00:57:23.982483 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:23.982459 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 00:57:23.984044 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:23.984020 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52226975_0a51_4129_a2d9_1db215e76dac.slice/crio-f45ceb1f07faa6905a020ed94e1ba2959472c79f64668267e17011639fc3b9f6 WatchSource:0}: Error finding container f45ceb1f07faa6905a020ed94e1ba2959472c79f64668267e17011639fc3b9f6: Status 404 returned error can't find the container with id f45ceb1f07faa6905a020ed94e1ba2959472c79f64668267e17011639fc3b9f6 Oct 30 00:57:24.147993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.147961 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:24.521981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.521941 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" event={"ID":"52226975-0a51-4129-a2d9-1db215e76dac","Type":"ContainerStarted","Data":"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7"} Oct 30 00:57:24.522447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.522010 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" event={"ID":"52226975-0a51-4129-a2d9-1db215e76dac","Type":"ContainerStarted","Data":"f45ceb1f07faa6905a020ed94e1ba2959472c79f64668267e17011639fc3b9f6"} Oct 30 00:57:24.522447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.522187 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:24.528129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.528093 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" event={"ID":"adbed9a1-0ed5-4042-8979-691eb3cb6f0b","Type":"ContainerStarted","Data":"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba"} Oct 30 00:57:24.528269 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.528143 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" event={"ID":"adbed9a1-0ed5-4042-8979-691eb3cb6f0b","Type":"ContainerStarted","Data":"6bfc484692dc25289fd35fa27c2f30c3369cd5f09256dca07628a1ceb5362076"} Oct 30 00:57:24.528452 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.528424 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:24.562429 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.562379 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" podStartSLOduration=1.562364898 podStartE2EDuration="1.562364898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:24.560548172 +0000 UTC m=+1496.800783286" watchObservedRunningTime="2026-10-30 00:57:24.562364898 +0000 UTC m=+1496.802600010" Oct 30 00:57:24.615433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:24.615376 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" podStartSLOduration=1.615355775 podStartE2EDuration="1.615355775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:24.608530559 +0000 UTC m=+1496.848765673" watchObservedRunningTime="2026-10-30 00:57:24.615355775 +0000 UTC m=+1496.855590889" Oct 30 00:57:25.856932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:25.856900 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:27.434153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.434119 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:57:27.543854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.543830 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:57:27.548184 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.548147 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:57:27.563108 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563067 5938 generic.go:356] "Generic (PLEG): container finished" podID="b3b3bb54-064e-4373-b226-08b02119b430" containerID="64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98" exitCode=0 Oct 30 00:57:27.563272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563124 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:57:27.563272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerDied","Data":"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98"} Oct 30 00:57:27.563272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563248 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" Oct 30 00:57:27.563448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563285 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt" event={"ID":"b3b3bb54-064e-4373-b226-08b02119b430","Type":"ContainerDied","Data":"d2faf8d77f097f084bae280abc852ed7883dafb5b7c4d117a7751a29f7f35c83"} Oct 30 00:57:27.563448 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.563318 5938 scope.go:122] "RemoveContainer" containerID="64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98" Oct 30 00:57:27.568228 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.568201 5938 generic.go:356] "Generic (PLEG): container finished" podID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerID="540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08" exitCode=0 Oct 30 00:57:27.568469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.568282 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:27.568469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.568297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" event={"ID":"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3","Type":"ContainerDied","Data":"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08"} Oct 30 00:57:27.568469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.568344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" event={"ID":"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3","Type":"ContainerDied","Data":"0caa808f847f429e9611501f523a2e5bc122a6461ef1170b474da9faa6ac6a6e"} Oct 30 00:57:27.568469 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.568396 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h" Oct 30 00:57:27.570248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.570219 5938 generic.go:356] "Generic (PLEG): container finished" podID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerID="9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8" exitCode=0 Oct 30 00:57:27.570468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.570447 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" event={"ID":"404212e5-c892-4aad-9dc0-49cdcd1bed75","Type":"ContainerDied","Data":"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8"} Oct 30 00:57:27.570553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.570493 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" event={"ID":"404212e5-c892-4aad-9dc0-49cdcd1bed75","Type":"ContainerDied","Data":"c971d3c374ffbad9731927e78559c1adefd1d81f3cea39ca2eb493604a5e347a"} Oct 30 00:57:27.570620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.570569 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn" Oct 30 00:57:27.575894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.575853 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.575997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.575915 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.575997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.575981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576053 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576123 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576227 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576188 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576228 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576315 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576280 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.576426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.576350 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\") pod \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\" (UID: \"f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3\") " Oct 30 00:57:27.577709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.577681 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:27.578244 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.578218 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579382 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579465 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579474 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579698 5938 generic.go:356] "Generic (PLEG): container finished" podID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerID="bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e" exitCode=0 Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579732 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" event={"ID":"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc","Type":"ContainerDied","Data":"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e"} Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579770 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" event={"ID":"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc","Type":"ContainerDied","Data":"8d78b4748208b83d7c8a13199b3d5b0b35233c1f16d1db737ed8f37bca400530"} Oct 30 00:57:27.580563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.579842 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw" Oct 30 00:57:27.584273 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.584250 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.584379 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.584288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "kube-api-access-n28bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.585323 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.585301 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.585441 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.585418 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo" pod "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" (UID: "f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:27.591053 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.591020 5938 scope.go:122] "RemoveContainer" containerID="4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600" Oct 30 00:57:27.603362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.603341 5938 scope.go:122] "RemoveContainer" containerID="64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98" Oct 30 00:57:27.603793 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:27.603646 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98\": container with ID starting with 64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98 not found: ID does not exist" containerID="64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98" Oct 30 00:57:27.603793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.603688 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98"} err="failed to get container status \"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98\": rpc error: code = NotFound desc = could not find container \"64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98\": container with ID starting with 64067b3ea870b8d2d82accef824457d3b749587baf4b487b59810cce87da8a98 not found: ID does not exist" Oct 30 00:57:27.603793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.603715 5938 scope.go:122] "RemoveContainer" containerID="4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600" Oct 30 00:57:27.604263 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:27.604233 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600\": container with ID starting with 4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600 not found: ID does not exist" containerID="4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600" Oct 30 00:57:27.604347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.604273 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600"} err="failed to get container status \"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600\": rpc error: code = NotFound desc = could not find container \"4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600\": container with ID starting with 4c39de12ff021b22bc25437ffb07644ce908874182508f27283093da3b724600 not found: ID does not exist" Oct 30 00:57:27.604347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.604295 5938 scope.go:122] "RemoveContainer" containerID="540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08" Oct 30 00:57:27.616492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.616419 5938 scope.go:122] "RemoveContainer" containerID="540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08" Oct 30 00:57:27.616797 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:27.616777 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08\": container with ID starting with 540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08 not found: ID does not exist" containerID="540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08" Oct 30 00:57:27.616912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.616803 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08"} err="failed to get container status \"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08\": rpc error: code = NotFound desc = could not find container \"540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08\": container with ID starting with 540d0b2144f6a998a182df4bbff875169f1babc1140430af91d2d5410745df08 not found: ID does not exist" Oct 30 00:57:27.616912 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.616820 5938 scope.go:122] "RemoveContainer" containerID="9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8" Oct 30 00:57:27.630389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.630366 5938 scope.go:122] "RemoveContainer" containerID="9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8" Oct 30 00:57:27.630704 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:27.630676 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8\": container with ID starting with 9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8 not found: ID does not exist" containerID="9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8" Oct 30 00:57:27.630805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.630718 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8"} err="failed to get container status \"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8\": rpc error: code = NotFound desc = could not find container \"9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8\": container with ID starting with 9e1b903474302e45facb00b1db2163d42f7e95d4e4d264da8e375acbd562ebf8 not found: ID does not exist" Oct 30 00:57:27.630805 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.630750 5938 scope.go:122] "RemoveContainer" containerID="bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e" Oct 30 00:57:27.645233 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.645215 5938 scope.go:122] "RemoveContainer" containerID="bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e" Oct 30 00:57:27.645548 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:27.645527 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e\": container with ID starting with bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e not found: ID does not exist" containerID="bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e" Oct 30 00:57:27.645648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.645562 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e"} err="failed to get container status \"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e\": rpc error: code = NotFound desc = could not find container \"bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e\": container with ID starting with bda8f3f8227646aa811c44a4f69554d66b6df4609af96bddf1c7cb110cf11d2e not found: ID does not exist" Oct 30 00:57:27.678507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.678507 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678561 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678599 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678662 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678695 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678724 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.678773 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678756 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.679680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678805 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.679680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678920 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.679680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.678981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.679680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.679016 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.679680 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.679452 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.679992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.679684 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.680077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680051 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.680151 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680135 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.680280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680190 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680237 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680289 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680316 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.680502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680398 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.680502 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680440 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.680619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680556 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.680619 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680599 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680644 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.680733 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\") pod \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\" (UID: \"1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc\") " Oct 30 00:57:27.680841 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680799 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\") pod \"404212e5-c892-4aad-9dc0-49cdcd1bed75\" (UID: \"404212e5-c892-4aad-9dc0-49cdcd1bed75\") " Oct 30 00:57:27.680961 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.680935 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\") pod \"b3b3bb54-064e-4373-b226-08b02119b430\" (UID: \"b3b3bb54-064e-4373-b226-08b02119b430\") " Oct 30 00:57:27.682340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682172 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682524 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682548 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682563 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682575 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682588 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28bp\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-kube-api-access-n28bp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682602 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682617 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682630 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682642 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682655 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682667 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682680 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.683058 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.682769 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.683392 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.683657 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684458 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684587 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684686 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684745 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684756 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.684826 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684781 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.685286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.684876 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.685286 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.685069 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.685403 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.685329 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:27.686515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.686472 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:57:27.688349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.688320 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "kube-api-access-588tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.693293 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.693258 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:27.693560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.693519 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "kube-api-access-vvmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.693663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.693577 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.693907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.693843 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.694094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.694069 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "kube-api-access-9kw8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:57:27.694201 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.694185 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy" pod "b3b3bb54-064e-4373-b226-08b02119b430" (UID: "b3b3bb54-064e-4373-b226-08b02119b430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.694276 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.694218 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:27.694344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.694275 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy" pod "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" (UID: "1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.694344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.694291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:57:27.695621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.695594 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo" pod "404212e5-c892-4aad-9dc0-49cdcd1bed75" (UID: "404212e5-c892-4aad-9dc0-49cdcd1bed75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:57:27.784316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784282 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404212e5-c892-4aad-9dc0-49cdcd1bed75-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784316 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784316 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784326 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784335 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784343 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784354 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784364 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3b3bb54-064e-4373-b226-08b02119b430-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784373 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784381 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784391 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kw8m\" (UniqueName: \"kubernetes.io/projected/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-kube-api-access-9kw8m\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784400 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmt7\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-kube-api-access-vvmt7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784411 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-588tf\" (UniqueName: \"kubernetes.io/projected/b3b3bb54-064e-4373-b226-08b02119b430-kube-api-access-588tf\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784419 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784429 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404212e5-c892-4aad-9dc0-49cdcd1bed75-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784437 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784447 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784455 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784462 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3b3bb54-064e-4373-b226-08b02119b430-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784470 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784477 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784484 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784492 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404212e5-c892-4aad-9dc0-49cdcd1bed75-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784499 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3b3bb54-064e-4373-b226-08b02119b430-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:27.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:27.784506 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:57:28.038660 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.038613 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:57:28.047078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.047050 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-qq9bt"] Oct 30 00:57:28.082882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.082830 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:57:28.087410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.087380 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-hhkvn"] Oct 30 00:57:28.122977 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.122953 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:57:28.128649 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.128623 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-c88jw"] Oct 30 00:57:28.158417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.158389 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:57:28.163765 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.163739 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-rc62h"] Oct 30 00:57:28.597982 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.597947 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:57:28.598433 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:28.597994 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:57:28.841163 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:28.841128 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:57:29.380978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.380943 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.599170 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.599136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.795131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.795096 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:29.879576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.879546 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" path="/var/lib/kubelet/pods/1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc/volumes" Oct 30 00:57:29.880200 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.880178 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" path="/var/lib/kubelet/pods/404212e5-c892-4aad-9dc0-49cdcd1bed75/volumes" Oct 30 00:57:29.880735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.880715 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b3bb54-064e-4373-b226-08b02119b430" path="/var/lib/kubelet/pods/b3b3bb54-064e-4373-b226-08b02119b430/volumes" Oct 30 00:57:29.881605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:29.881585 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" path="/var/lib/kubelet/pods/f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3/volumes" Oct 30 00:57:30.008240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.008174 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:30.120886 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:30.113961 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:30.390625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.390548 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:57:30.418608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.418587 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 00:57:30.503382 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.503359 5938 scope.go:122] "RemoveContainer" containerID="b88a53e46053f467c45fdd70d44e76dc33bacc26c41effb6225971b7a0a667a0" Oct 30 00:57:30.513077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.513061 5938 scope.go:122] "RemoveContainer" containerID="fed0f80a2433ba4568f29ac0242d94700396beb7c0b440c2f6be6501ddbf588f" Oct 30 00:57:30.522835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.522814 5938 scope.go:122] "RemoveContainer" containerID="239cc0dc3f337722bac3712621c6da21f6e878a53439076e4e19574c8bf3d727" Oct 30 00:57:30.531857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.531839 5938 scope.go:122] "RemoveContainer" containerID="f8d02666d31e281a7d18d6a4de650e9e1ee55c8ca146b5cce8c7a697dbe01dc5" Oct 30 00:57:30.610486 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.610463 5938 generic.go:356] "Generic (PLEG): container finished" podID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerID="613c85d3764687b299cea8049862bb9e561ef82d060b69c13a5773a112410193" exitCode=0 Oct 30 00:57:30.610879 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.610531 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerDied","Data":"613c85d3764687b299cea8049862bb9e561ef82d060b69c13a5773a112410193"} Oct 30 00:57:30.610978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.610963 5938 scope.go:122] "RemoveContainer" containerID="613c85d3764687b299cea8049862bb9e561ef82d060b69c13a5773a112410193" Oct 30 00:57:30.612638 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.612604 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:30.612753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.612691 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:30.612823 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.612749 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:30.876911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:30.876841 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:31.267839 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.267800 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:31.422523 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.422484 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 00:57:31.423278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423260 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" Oct 30 00:57:31.423278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423277 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423293 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423298 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423307 5938 cpu_manager.go:414] "containerMap: removing container" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423313 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423329 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423333 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423460 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ef6ce5-829f-4a9b-bd63-8b14a8c7e2b3" containerName="istio-proxy" Oct 30 00:57:31.423463 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423467 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:31.423925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423473 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="404212e5-c892-4aad-9dc0-49cdcd1bed75" containerName="istio-proxy" Oct 30 00:57:31.423925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423481 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:31.423925 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.423488 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce8ba35-7602-4ebb-b1d6-8a0fbe5f35cc" containerName="istio-proxy" Oct 30 00:57:31.426343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.426320 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.451191 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.451162 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 00:57:31.460079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.460046 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d2a70-openshift-default-dockercfg-vpjnj\"" Oct 30 00:57:31.510963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.510918 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfts\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511014 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511053 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511105 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511177 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511154 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511232 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511285 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511315 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.511514 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.511460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.530419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.530346 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:31.612465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612433 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612550 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfts\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612632 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612675 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612727 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612778 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612876 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612938 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613035 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.612970 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.613234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.613244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613367 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.613330 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613492 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.613476 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.613990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.613962 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.615007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.614988 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.616110 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.616093 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.622598 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.622574 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerStarted","Data":"d676b1a80983773d0750bcf9fa89df0ca64e5987a9e20ffbc66ef4b98f49fce9"} Oct 30 00:57:31.636231 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.636206 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.637385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.637364 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfts\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:31.653883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.642855 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:57:31.653883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.642998 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:31.653883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.643037 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:31.689898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.686454 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:31.760990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:31.760189 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:32.212654 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:32.212614 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f729a8b_c623_4934_a3b7_e50288a049ce.slice/crio-5d247b6ceb32b3a02b0c369b792923ec0e5bb1aaaf11ae9f7eb77cc48fd858a5 WatchSource:0}: Error finding container 5d247b6ceb32b3a02b0c369b792923ec0e5bb1aaaf11ae9f7eb77cc48fd858a5: Status 404 returned error can't find the container with id 5d247b6ceb32b3a02b0c369b792923ec0e5bb1aaaf11ae9f7eb77cc48fd858a5 Oct 30 00:57:32.214074 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.214040 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 00:57:32.216290 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.216255 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:32.216398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.216348 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:32.216398 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.216383 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:32.632611 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.632577 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerStarted","Data":"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005"} Oct 30 00:57:32.633103 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.632635 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerStarted","Data":"5d247b6ceb32b3a02b0c369b792923ec0e5bb1aaaf11ae9f7eb77cc48fd858a5"} Oct 30 00:57:32.642480 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.642456 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:32.642603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.642507 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:32.699886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.697814 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podStartSLOduration=1.697795842 podStartE2EDuration="1.697795842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:32.689273041 +0000 UTC m=+1504.929508155" watchObservedRunningTime="2026-10-30 00:57:32.697795842 +0000 UTC m=+1504.938030958" Oct 30 00:57:32.762538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:32.762503 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:33.104942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:33.104295 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:33.255894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:33.249162 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:33.642044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:33.642015 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:33.642454 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:33.642073 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:33.769101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:33.769067 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:34.642533 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:34.642499 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:34.642939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:34.642549 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:34.650522 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:34.650496 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:34.654120 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:34.654091 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:57:34.745092 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:34.745053 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:35.099287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.099205 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:35.256804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.256761 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:35.433271 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.433223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:35.539666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.539638 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 00:57:35.539845 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.539711 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 00:57:35.642307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.642277 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:35.642535 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:35.642509 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:36.641817 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:36.641782 5938 patch_prober.go:29] interesting pod/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 00:57:36.642007 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:36.641831 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 00:57:36.992815 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:36.992730 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:37.058652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.058614 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 00:57:37.059785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.059760 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:37.059785 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.059781 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b3bb54-064e-4373-b226-08b02119b430" containerName="istio-proxy" Oct 30 00:57:37.064294 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.064272 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.078557 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.078528 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 00:57:37.084465 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.084442 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8d2a70\"" Oct 30 00:57:37.103992 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.103962 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 00:57:37.107877 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.107849 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.124082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.124060 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 00:57:37.125716 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.125690 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8d2a70-dockercfg-2cmlq\"" Oct 30 00:57:37.132825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.132797 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.133008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.132986 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmg6\" (UniqueName: \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.136277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.136255 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:37.346575 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.346478 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfrc\" (UniqueName: \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.346769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.346612 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.346769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.346757 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.346947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.346792 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmg6\" (UniqueName: \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.347651 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.347617 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.369768 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.369739 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmg6\" (UniqueName: \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\") pod \"payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.387949 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.387925 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:37.452946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.450941 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfrc\" (UniqueName: \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.452946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.451097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.452946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.452123 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.481067 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.476148 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfrc\" (UniqueName: \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\") pod \"payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.506888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.504621 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:37.646453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.646425 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:57:37.646781 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.646756 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:57:37.647602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.647582 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 00:57:37.698615 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:37.698589 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdb22ff_c281_42aa_b35f_e5a272594283.slice/crio-b71dbac305bccd8f456e917d2befae3276dcd59fb10359a57838d75718518308 WatchSource:0}: Error finding container b71dbac305bccd8f456e917d2befae3276dcd59fb10359a57838d75718518308: Status 404 returned error can't find the container with id b71dbac305bccd8f456e917d2befae3276dcd59fb10359a57838d75718518308 Oct 30 00:57:37.698830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.698810 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 00:57:37.730314 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.730291 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:37.940565 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:37.940494 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:38.050628 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.050602 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 00:57:38.052214 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:38.052188 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04a828a_6b01_48a3_bf19_fed27ae4399a.slice/crio-fbccecdb221879e2adb5637d91c05f2bc7991a67e6a7d4767eccb8e5900b6aaf WatchSource:0}: Error finding container fbccecdb221879e2adb5637d91c05f2bc7991a67e6a7d4767eccb8e5900b6aaf: Status 404 returned error can't find the container with id fbccecdb221879e2adb5637d91c05f2bc7991a67e6a7d4767eccb8e5900b6aaf Oct 30 00:57:38.685359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.685317 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" event={"ID":"b04a828a-6b01-48a3-bf19-fed27ae4399a","Type":"ContainerStarted","Data":"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418"} Oct 30 00:57:38.685532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.685379 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:38.685532 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.685402 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" event={"ID":"b04a828a-6b01-48a3-bf19-fed27ae4399a","Type":"ContainerStarted","Data":"fbccecdb221879e2adb5637d91c05f2bc7991a67e6a7d4767eccb8e5900b6aaf"} Oct 30 00:57:38.687050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.687024 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" event={"ID":"8fdb22ff-c281-42aa-b35f-e5a272594283","Type":"ContainerStarted","Data":"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b"} Oct 30 00:57:38.687167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.687116 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:38.687167 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.687157 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" event={"ID":"8fdb22ff-c281-42aa-b35f-e5a272594283","Type":"ContainerStarted","Data":"b71dbac305bccd8f456e917d2befae3276dcd59fb10359a57838d75718518308"} Oct 30 00:57:38.727626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.727582 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" podStartSLOduration=1.727570197 podStartE2EDuration="1.727570197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:38.723474851 +0000 UTC m=+1510.963709965" watchObservedRunningTime="2026-10-30 00:57:38.727570197 +0000 UTC m=+1510.967805361" Oct 30 00:57:38.764554 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:38.764513 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" podStartSLOduration=1.764499366 podStartE2EDuration="1.764499366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:38.76117029 +0000 UTC m=+1511.001405403" watchObservedRunningTime="2026-10-30 00:57:38.764499366 +0000 UTC m=+1511.004734492" Oct 30 00:57:39.292017 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:39.291986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:41.067188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.067153 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 00:57:41.071321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.071303 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.094131 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.094102 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 00:57:41.097793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.097762 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:41.099672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.099648 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6444d7-openshift-default-dockercfg-84h8h\"" Oct 30 00:57:41.208117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208084 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208328 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208400 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208366 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208566 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208549 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllcp\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208606 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208581 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208643 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208694 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208674 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208733 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.208793 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.208757 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.241451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.241417 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:41.310347 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310321 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllcp\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310356 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310396 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310432 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310478 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310508 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310676 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.310807 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310736 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.311082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310885 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.311082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.310969 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.311203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.311143 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.311385 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.311369 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.313026 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.312995 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.313143 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.312994 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.334310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.334256 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.335214 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.335189 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllcp\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\") pod \"e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.386029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.386008 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:41.430927 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.430894 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:41.705215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.705185 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 00:57:41.705727 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:41.705702 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d5ec8a_460c_4bdd_a361_5cb14bd4537d.slice/crio-faef4b6534c01495391ce980888dc0ab3becd790a6bc56afdcc42ad62fcb7633 WatchSource:0}: Error finding container faef4b6534c01495391ce980888dc0ab3becd790a6bc56afdcc42ad62fcb7633: Status 404 returned error can't find the container with id faef4b6534c01495391ce980888dc0ab3becd790a6bc56afdcc42ad62fcb7633 Oct 30 00:57:41.708349 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.708317 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:41.708450 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.708380 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:41.708450 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.708413 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:41.717505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:41.717476 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" event={"ID":"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d","Type":"ContainerStarted","Data":"faef4b6534c01495391ce980888dc0ab3becd790a6bc56afdcc42ad62fcb7633"} Oct 30 00:57:42.726871 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:42.726808 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" event={"ID":"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d","Type":"ContainerStarted","Data":"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38"} Oct 30 00:57:42.813691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:42.813636 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podStartSLOduration=1.813621323 podStartE2EDuration="1.813621323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:42.810028102 +0000 UTC m=+1515.050263214" watchObservedRunningTime="2026-10-30 00:57:42.813621323 +0000 UTC m=+1515.053856511" Oct 30 00:57:43.409187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:43.409157 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:43.410594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:43.410566 5938 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 00:57:43.410734 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:43.410623 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 00:57:43.413197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:43.413143 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:44.009666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:44.009623 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:44.409247 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:44.409212 5938 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 00:57:44.409462 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:44.409310 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 00:57:44.859479 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:44.859452 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:44.971696 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:44.971661 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:45.106889 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:45.098437 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:45.406888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:45.399162 5938 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 00:57:45.406888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:45.399226 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 00:57:45.426639 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:45.424128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:46.408889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:46.398407 5938 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 00:57:46.410208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:46.410171 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 00:57:47.314888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:47.313656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:47.398888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:47.390512 5938 patch_prober.go:29] interesting pod/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 00:57:47.398888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:47.390581 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 00:57:48.391600 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:48.391570 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:48.392117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:48.392050 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:48.392825 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:48.392808 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 00:57:49.095905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:49.095847 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:49.697490 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:49.697465 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 00:57:49.697950 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:49.697873 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 00:57:50.804031 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:50.803998 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:52.684835 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:52.684786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:54.950585 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:54.950555 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:57:56.655042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:56.655007 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:57:56.684590 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:56.684562 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:57:56.892757 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:56.892705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:57.054620 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:57.054531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:57.197657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:57.197629 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:57.692827 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:57.692793 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:58.124101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.124068 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 00:57:58.128544 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.128521 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.158709 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.158684 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 00:57:58.187614 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.187585 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-03d892-openshift-default-dockercfg-crhvb\"" Oct 30 00:57:58.221323 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.221279 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:58.234310 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234281 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234334 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234529 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234482 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234598 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234794 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234766 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234800 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkh8\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.234926 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234824 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.235006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.234948 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.235057 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.235043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363003 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363048 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkh8\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363079 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363109 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363201 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363292 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363333 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.363499 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.364020 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.373889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.364828 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.383015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.382633 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:58.394884 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.388273 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.423282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.399216 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.423282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.399671 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.423282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.405338 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.423282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.417203 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.463887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.463643 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkh8\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.488888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.488419 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\") pod \"e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.758330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.758246 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:57:58.761226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.761191 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:58.936101 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:57:58.921478 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:57:58.936101 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:58.932570 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:59.095563 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.095532 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 00:57:59.096559 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:57:59.096532 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b0d383_83ea_444e_903f_d50785175255.slice/crio-348747f54861c1dacf41f7689577e52bbef0f8c326730080650a39e32a54ad76 WatchSource:0}: Error finding container 348747f54861c1dacf41f7689577e52bbef0f8c326730080650a39e32a54ad76: Status 404 returned error can't find the container with id 348747f54861c1dacf41f7689577e52bbef0f8c326730080650a39e32a54ad76 Oct 30 00:57:59.100374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.099801 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:59.100374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.099912 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:59.100374 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.099955 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:57:59.205911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.205215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:59.717122 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:57:59.715684 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:00.050695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.025711 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerStarted","Data":"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db"} Oct 30 00:58:00.050695 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.049336 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerStarted","Data":"348747f54861c1dacf41f7689577e52bbef0f8c326730080650a39e32a54ad76"} Oct 30 00:58:00.112890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.112398 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podStartSLOduration=2.112378212 podStartE2EDuration="2.112378212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:00.101490808 +0000 UTC m=+1532.341725921" watchObservedRunningTime="2026-10-30 00:58:00.112378212 +0000 UTC m=+1532.352613326" Oct 30 00:58:00.233834 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:00.233793 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:00.312093 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.311995 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:00.480748 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.480704 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:00.760029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.759783 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:58:00.773920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:00.773263 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:58:01.067239 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:01.067147 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:58:01.068914 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:01.068245 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:58:01.150857 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:01.150819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:01.657738 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:01.656917 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" podUID="20feefc3-1f0b-456f-a672-b98a21d065f7" containerName="payload-processing" containerID="cri-o://460c3202dd3d1c025a4be3f35f1c578aba64110b77c42af5e607bf94079fe444" gracePeriod=25 Oct 30 00:58:01.858888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:01.855061 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" podUID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" containerName="payload-pre-processing" containerID="cri-o://0ea2706e7e321484052cbaa1807680f5abf5684be1d8727e6d7ba0815025c0c1" gracePeriod=25 Oct 30 00:58:02.564941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.561186 5938 generic.go:356] "Generic (PLEG): container finished" podID="20feefc3-1f0b-456f-a672-b98a21d065f7" containerID="460c3202dd3d1c025a4be3f35f1c578aba64110b77c42af5e607bf94079fe444" exitCode=0 Oct 30 00:58:02.564941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.561316 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" event={"ID":"20feefc3-1f0b-456f-a672-b98a21d065f7","Type":"ContainerDied","Data":"460c3202dd3d1c025a4be3f35f1c578aba64110b77c42af5e607bf94079fe444"} Oct 30 00:58:02.568008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.566926 5938 generic.go:356] "Generic (PLEG): container finished" podID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" containerID="0ea2706e7e321484052cbaa1807680f5abf5684be1d8727e6d7ba0815025c0c1" exitCode=0 Oct 30 00:58:02.568008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.567941 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" event={"ID":"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b","Type":"ContainerDied","Data":"0ea2706e7e321484052cbaa1807680f5abf5684be1d8727e6d7ba0815025c0c1"} Oct 30 00:58:02.758885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.748074 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:02.758885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.755045 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:58:02.921548 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.919929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:02.968136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.968073 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\") pod \"20feefc3-1f0b-456f-a672-b98a21d065f7\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " Oct 30 00:58:02.969149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.969118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\" (UniqueName: \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\") pod \"20feefc3-1f0b-456f-a672-b98a21d065f7\" (UID: \"20feefc3-1f0b-456f-a672-b98a21d065f7\") " Oct 30 00:58:02.970104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.969005 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume" pod "20feefc3-1f0b-456f-a672-b98a21d065f7" (UID: "20feefc3-1f0b-456f-a672-b98a21d065f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:02.976811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:02.976770 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs" pod "20feefc3-1f0b-456f-a672-b98a21d065f7" (UID: "20feefc3-1f0b-456f-a672-b98a21d065f7"). InnerVolumeSpecName "kube-api-access-wn7vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:03.070911 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.059912 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw"] Oct 30 00:58:03.203939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.195169 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:03.203939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.196938 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f0071b-cxhkw"] Oct 30 00:58:03.219903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.217756 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:03.264436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.264396 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20feefc3-1f0b-456f-a672-b98a21d065f7-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:03.264668 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.264654 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7vs\" (UniqueName: \"kubernetes.io/projected/20feefc3-1f0b-456f-a672-b98a21d065f7-kube-api-access-wn7vs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:03.295832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.273687 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:58:03.365426 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.365385 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\") pod \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " Oct 30 00:58:03.367015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.366988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\" (UniqueName: \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\") pod \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\" (UID: \"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b\") " Oct 30 00:58:03.368082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.366619 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume" pod "bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" (UID: "bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:03.369148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.368980 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:03.465252 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.465144 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": context deadline exceeded" Oct 30 00:58:03.497153 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.497095 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms" pod "bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" (UID: "bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b"). InnerVolumeSpecName "kube-api-access-mnvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:03.580710 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.580678 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnvms\" (UniqueName: \"kubernetes.io/projected/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b-kube-api-access-mnvms\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:03.674714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.668896 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" Oct 30 00:58:03.674714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.668896 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp" event={"ID":"20feefc3-1f0b-456f-a672-b98a21d065f7","Type":"ContainerDied","Data":"df4c4f7eafcf10ced17f0563c9516027c8a70968c135fdfa85d08c9066ad2198"} Oct 30 00:58:03.674714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.669066 5938 scope.go:122] "RemoveContainer" containerID="460c3202dd3d1c025a4be3f35f1c578aba64110b77c42af5e607bf94079fe444" Oct 30 00:58:03.674714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.671631 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" Oct 30 00:58:03.674714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.671663 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b" event={"ID":"bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b","Type":"ContainerDied","Data":"2f5255c3b446db55e6bebaa1391d630bb705e87f1199b6ba89e2405ae8153814"} Oct 30 00:58:03.683918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.683849 5938 scope.go:122] "RemoveContainer" containerID="0ea2706e7e321484052cbaa1807680f5abf5684be1d8727e6d7ba0815025c0c1" Oct 30 00:58:03.770087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.769937 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:58:03.773612 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.773582 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fa8dd0-9dbb6d4dd-cbtwp"] Oct 30 00:58:03.818944 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.818911 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:58:03.843940 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.840377 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fa8dd0-77bcf7f86c-rzx8b"] Oct 30 00:58:03.879284 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.879253 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20feefc3-1f0b-456f-a672-b98a21d065f7" path="/var/lib/kubelet/pods/20feefc3-1f0b-456f-a672-b98a21d065f7/volumes" Oct 30 00:58:03.879721 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.879704 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" path="/var/lib/kubelet/pods/bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b/volumes" Oct 30 00:58:03.880209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:03.880193 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf51dfce-b0d5-49de-abb2-53c6fe009d6e" path="/var/lib/kubelet/pods/bf51dfce-b0d5-49de-abb2-53c6fe009d6e/volumes" Oct 30 00:58:04.377165 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.376639 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:04.569654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.569619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:04.581997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.581964 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 00:58:04.583682 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.583652 5938 cpu_manager.go:414] "containerMap: removing container" podUID="20feefc3-1f0b-456f-a672-b98a21d065f7" containerName="payload-processing" Oct 30 00:58:04.583897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.583854 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20feefc3-1f0b-456f-a672-b98a21d065f7" containerName="payload-processing" Oct 30 00:58:04.584050 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.584037 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" containerName="payload-pre-processing" Oct 30 00:58:04.584147 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.584134 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" containerName="payload-pre-processing" Oct 30 00:58:04.605084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.605051 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="20feefc3-1f0b-456f-a672-b98a21d065f7" containerName="payload-processing" Oct 30 00:58:04.632521 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.632436 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3e73c7-dc8f-4c85-8a7c-f8f907a63d6b" containerName="payload-pre-processing" Oct 30 00:58:04.729888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.726810 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:04.879887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.865283 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 00:58:04.883942 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.880623 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-03d892\"" Oct 30 00:58:04.947888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.947565 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:04.947888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:04.947805 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-03d892-dockercfg-m84cc\"" Oct 30 00:58:05.009471 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.009437 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.013883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.009896 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr59n\" (UniqueName: \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.063887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.040693 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 00:58:05.063887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.041244 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.075920 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.066391 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 00:58:05.163937 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:05.097263 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:05.175118 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.175083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:05.177241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.177209 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr59n\" (UniqueName: \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.177674 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.177647 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.177983 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.177904 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz654\" (UniqueName: \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.178447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.178371 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.183592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.183542 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.275980 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.275283 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr59n\" (UniqueName: \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\") pod \"payload-processing-e2e-inf-b-03d892-696b99c46-mb79k\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.280951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.280918 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.281174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.281158 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz654\" (UniqueName: \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.281704 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.281602 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.341320 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.341289 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:05.350854 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.350821 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz654\" (UniqueName: \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\") pod \"payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:05.484449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.469704 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:05.488517 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:05.478226 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:06.271188 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.271143 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 00:58:06.278960 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:06.278933 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9ad999_ac50_491f_9af1_a51a76ed08c2.slice/crio-d1241f41cb63e489baab5448d5e91ab1d60c71af68b7a5b641cedd6f277c140f WatchSource:0}: Error finding container d1241f41cb63e489baab5448d5e91ab1d60c71af68b7a5b641cedd6f277c140f: Status 404 returned error can't find the container with id d1241f41cb63e489baab5448d5e91ab1d60c71af68b7a5b641cedd6f277c140f Oct 30 00:58:06.570633 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.563565 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 00:58:06.570633 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:06.566272 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6b9f4a_72b6_45ed_8d08_f8ed9244a633.slice/crio-fa613ac6979d8e8ea4db9f8aedec60db24fd3e505603d33f953bcb3de96cbf2c WatchSource:0}: Error finding container fa613ac6979d8e8ea4db9f8aedec60db24fd3e505603d33f953bcb3de96cbf2c: Status 404 returned error can't find the container with id fa613ac6979d8e8ea4db9f8aedec60db24fd3e505603d33f953bcb3de96cbf2c Oct 30 00:58:06.631407 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.631363 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:06.683027 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.682993 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 00:58:06.688707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.688667 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.707910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.707873 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 00:58:06.715078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.715041 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-84b89f-dockercfg-zqfvw\"" Oct 30 00:58:06.715270 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.715107 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-84b89f\"" Oct 30 00:58:06.745909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.745877 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 00:58:06.751365 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.751341 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.772788 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.772394 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 00:58:06.778899 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.778837 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:06.844136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.838401 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbmk\" (UniqueName: \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.844136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.838463 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.844136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.838589 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964rm\" (UniqueName: \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.844136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.838706 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.955617 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbmk\" (UniqueName: \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.955691 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.955899 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964rm\" (UniqueName: \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.956077 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.962290 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:06.974268 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.968956 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.995410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.989829 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbmk\" (UniqueName: \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\") pod \"payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:06.995410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:06.993810 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-964rm\" (UniqueName: \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\") pod \"payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:07.031558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.031524 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:07.072258 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.072225 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:07.211786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.211502 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" event={"ID":"fb9ad999-ac50-491f-9af1-a51a76ed08c2","Type":"ContainerStarted","Data":"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e"} Oct 30 00:58:07.211786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.211565 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" event={"ID":"fb9ad999-ac50-491f-9af1-a51a76ed08c2","Type":"ContainerStarted","Data":"d1241f41cb63e489baab5448d5e91ab1d60c71af68b7a5b641cedd6f277c140f"} Oct 30 00:58:07.212032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.211807 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:07.216558 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.216528 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" event={"ID":"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633","Type":"ContainerStarted","Data":"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186"} Oct 30 00:58:07.216691 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.216581 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" event={"ID":"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633","Type":"ContainerStarted","Data":"fa613ac6979d8e8ea4db9f8aedec60db24fd3e505603d33f953bcb3de96cbf2c"} Oct 30 00:58:07.217353 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.217327 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:07.264621 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.259371 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" podStartSLOduration=3.259353898 podStartE2EDuration="3.259353898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:07.253213542 +0000 UTC m=+1539.493448656" watchObservedRunningTime="2026-10-30 00:58:07.259353898 +0000 UTC m=+1539.499589012" Oct 30 00:58:07.301137 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.298098 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" podStartSLOduration=3.298080742 podStartE2EDuration="3.298080742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:07.292419598 +0000 UTC m=+1539.532654712" watchObservedRunningTime="2026-10-30 00:58:07.298080742 +0000 UTC m=+1539.538315856" Oct 30 00:58:07.363832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.363796 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 00:58:07.364684 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:07.364651 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24cbb2f_fe92_429f_b933_53ab35ee1c9d.slice/crio-901b489ba32e1b36126b22e4862b20fd349e4c4c1ee1b6b7026f30412ca55ad7 WatchSource:0}: Error finding container 901b489ba32e1b36126b22e4862b20fd349e4c4c1ee1b6b7026f30412ca55ad7: Status 404 returned error can't find the container with id 901b489ba32e1b36126b22e4862b20fd349e4c4c1ee1b6b7026f30412ca55ad7 Oct 30 00:58:07.404024 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.403228 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 00:58:07.821963 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:07.816220 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:08.229799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.229765 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" event={"ID":"82c08318-dfc9-4519-8e78-dc4cd767a627","Type":"ContainerStarted","Data":"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf"} Oct 30 00:58:08.230096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.230073 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" event={"ID":"82c08318-dfc9-4519-8e78-dc4cd767a627","Type":"ContainerStarted","Data":"3e261c4082bc04afec12034a4067861d1e9491b54c51faf89e9138b7407ff60d"} Oct 30 00:58:08.230248 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.230234 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:08.232321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.232297 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" event={"ID":"c24cbb2f-fe92-429f-b933-53ab35ee1c9d","Type":"ContainerStarted","Data":"ee0fe5931b49605b8dd413b4cf13e68f97d4aa6a964c65aae2ebca7880fdbb77"} Oct 30 00:58:08.232435 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.232344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" event={"ID":"c24cbb2f-fe92-429f-b933-53ab35ee1c9d","Type":"ContainerStarted","Data":"901b489ba32e1b36126b22e4862b20fd349e4c4c1ee1b6b7026f30412ca55ad7"} Oct 30 00:58:08.232705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.232691 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:08.274416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.274361 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" podStartSLOduration=2.274345661 podStartE2EDuration="2.274345661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:08.268372226 +0000 UTC m=+1540.508607338" watchObservedRunningTime="2026-10-30 00:58:08.274345661 +0000 UTC m=+1540.514580800" Oct 30 00:58:08.319752 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:08.319688 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" podStartSLOduration=2.319665879 podStartE2EDuration="2.319665879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:08.312643611 +0000 UTC m=+1540.552878730" watchObservedRunningTime="2026-10-30 00:58:08.319665879 +0000 UTC m=+1540.559900995" Oct 30 00:58:09.702904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:09.702262 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:10.796010 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:10.795966 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:10.796470 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:10.796034 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:11.307652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:11.307616 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:11.763041 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:11.762977 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:11.763313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:11.763038 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:12.682272 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:12.682225 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:12.682272 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:12.682264 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:13.152813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:13.152768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:13.320203 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:13.319689 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:14.946044 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:14.946013 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:15.095027 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:15.094993 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:15.285684 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:15.285604 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:15.374625 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.374595 5938 generic.go:356] "Generic (PLEG): container finished" podID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerID="0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100" exitCode=0 Oct 30 00:58:15.374787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.374666 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerDied","Data":"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100"} Oct 30 00:58:15.375096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.375076 5938 scope.go:122] "RemoveContainer" containerID="0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100" Oct 30 00:58:15.376643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.376613 5938 generic.go:356] "Generic (PLEG): container finished" podID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerID="85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596" exitCode=0 Oct 30 00:58:15.376758 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.376691 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerDied","Data":"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596"} Oct 30 00:58:15.377020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.376992 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.377117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.377070 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.377117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.377101 5938 scope.go:122] "RemoveContainer" containerID="85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596" Oct 30 00:58:15.377241 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.377119 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.378830 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.378803 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.378932 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.378890 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.378979 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.378932 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.379266 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.379245 5938 generic.go:356] "Generic (PLEG): container finished" podID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerID="69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83" exitCode=0 Oct 30 00:58:15.379330 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.379272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerDied","Data":"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83"} Oct 30 00:58:15.379596 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.379579 5938 scope.go:122] "RemoveContainer" containerID="69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83" Oct 30 00:58:15.381206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.381178 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.381324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.381251 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:15.381324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:15.381300 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:16.441889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.429967 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerStarted","Data":"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4"} Oct 30 00:58:16.448902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.446020 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerStarted","Data":"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0"} Oct 30 00:58:16.465946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.463399 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerStarted","Data":"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a"} Oct 30 00:58:16.565511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.565470 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:58:16.565699 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.565656 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:16.565754 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.565705 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:16.762444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.762103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:16.936893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.936270 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:58:16.936893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.936396 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:16.936893 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:16.936453 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:17.442888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.442359 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:58:17.444934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.444431 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:58:17.444934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.444479 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 00:58:17.570114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.569995 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:17.570114 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.570071 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:17.947891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.947402 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:17.947891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:17.947465 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:18.146296 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.146037 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:58:18.260953 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.259036 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 00:58:18.479889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.478158 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:58:18.479889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.478238 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 00:58:18.571946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.571231 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:18.571946 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.571363 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:18.634918 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.626482 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:18.938890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.936311 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:18.938890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:18.936367 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:19.272939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.272527 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 00:58:19.276882 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.275297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 00:58:19.277033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.276924 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 00:58:19.671941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.576023 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:58:19.671941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.576096 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 00:58:19.671941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.578594 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:19.671941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.578640 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:19.966902 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.966797 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:19.967085 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:19.966896 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:20.333505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.333414 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:20.459029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.458992 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:58:20.459313 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.459279 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 00:58:20.566002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.565944 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:20.566174 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.566021 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:20.933737 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.933705 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:20.957208 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:20.957169 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:21.445331 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.445288 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 00:58:21.445636 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.445606 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 00:58:21.566700 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.566663 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 00:58:21.566905 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.566734 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 00:58:21.939428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.935431 5938 patch_prober.go:29] interesting pod/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 00:58:21.939428 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:21.935522 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 00:58:22.079769 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.079726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:22.444443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.444410 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:58:22.445681 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.445657 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 00:58:22.571344 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.571312 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:58:22.571889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.571854 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:58:22.572648 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.572626 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 00:58:22.937159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.937131 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:58:22.937443 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.937419 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:58:22.938034 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:22.938016 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 00:58:23.954745 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:23.954715 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:25.377759 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:25.377731 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404212e5_c892_4aad_9dc0_49cdcd1bed75.slice\": RecentStats: unable to find data in memory cache]" Oct 30 00:58:25.856720 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:25.856679 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:26.847352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:26.847307 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:27.293205 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.293174 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:27.612447 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.612419 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:27.705659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.705625 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 00:58:27.709340 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.709325 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.738543 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.738517 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce513-openshift-default-dockercfg-7qp6x\"" Oct 30 00:58:27.739002 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.738985 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 00:58:27.788923 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.788895 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789071 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789021 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789066 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psf8h\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789108 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789299 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789276 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789307 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789449 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789398 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789515 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789447 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.789576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.789557 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.804770 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.804746 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:27.912152 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.912050 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.947990 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.947949 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.948274 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.948256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.948427 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.948410 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.948609 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.948592 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.948836 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.948821 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.951297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.951264 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.951574 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.951549 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psf8h\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.951746 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.951729 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:27.955525 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.953082 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.953172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.953399 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.914415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.953611 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.956603 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.008886 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:27.971027 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.010278 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.010009 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psf8h\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.044803 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.044769 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.062581 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.062531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:28.340813 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.340736 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce513-openshift-default-dockercfg-7qp6x\"" Oct 30 00:58:28.348209 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.348182 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:28.685359 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.685331 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 00:58:28.687491 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:28.687468 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd77300_19e2_4629_8e82_5498fbcabb5b.slice/crio-b24c8490fcc4dcfab0df97f3a294bbd59e7e9d8293d69711706d428e7ddaafca WatchSource:0}: Error finding container b24c8490fcc4dcfab0df97f3a294bbd59e7e9d8293d69711706d428e7ddaafca: Status 404 returned error can't find the container with id b24c8490fcc4dcfab0df97f3a294bbd59e7e9d8293d69711706d428e7ddaafca Oct 30 00:58:28.690511 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.690477 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:28.690627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.690557 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:28.690627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.690608 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:28.795735 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.795698 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerStarted","Data":"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6"} Oct 30 00:58:28.795897 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.795774 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerStarted","Data":"b24c8490fcc4dcfab0df97f3a294bbd59e7e9d8293d69711706d428e7ddaafca"} Oct 30 00:58:28.843885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:28.843624 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podStartSLOduration=1.843609118 podStartE2EDuration="1.843609118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:28.841203264 +0000 UTC m=+1561.081438389" watchObservedRunningTime="2026-10-30 00:58:28.843609118 +0000 UTC m=+1561.083844230" Oct 30 00:58:28.850907 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:28.844622 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:58:29.348652 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:29.348619 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:29.354705 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:29.354681 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:29.392876 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:29.392838 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:29.804352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:29.804314 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:29.805246 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:29.805224 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 00:58:30.493711 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:30.493674 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:30.637287 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:30.637262 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:31.337256 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:31.337217 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:31.560714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:31.560682 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.039800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.039766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.178446 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.178421 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 00:58:33.193951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.192395 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.195297 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.195263 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.202055 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.202028 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 00:58:33.216508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.216483 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-cce513\"" Oct 30 00:58:33.216730 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.216711 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-cce513-dockercfg-bnrp9\"" Oct 30 00:58:33.242042 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.242013 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmfd\" (UniqueName: \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.242173 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.242093 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.284583 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.284557 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 00:58:33.289283 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.289236 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.308187 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.308161 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 00:58:33.469892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.435567 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.470547 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.439562 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmfd\" (UniqueName: \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.470800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.470776 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.483941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.470987 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.484325 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.484299 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw87\" (UniqueName: \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.504722 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.482800 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.549910 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.549021 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmfd\" (UniqueName: \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\") pod \"payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.606312 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.606281 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.606495 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.606428 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw87\" (UniqueName: \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.607107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.607087 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.629345 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.629315 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw87\" (UniqueName: \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\") pod \"payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:33.824015 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.823989 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:33.916052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:33.916023 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:34.131955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.131851 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 00:58:34.132311 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:34.132235 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6128a092_1455_4238_b4d6_57966665f1e3.slice/crio-a05dc89ce92447245565e11ccb26d1c4fb8b2aca6303c7d8306d5f159ea54fc3 WatchSource:0}: Error finding container a05dc89ce92447245565e11ccb26d1c4fb8b2aca6303c7d8306d5f159ea54fc3: Status 404 returned error can't find the container with id a05dc89ce92447245565e11ccb26d1c4fb8b2aca6303c7d8306d5f159ea54fc3 Oct 30 00:58:34.230460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.230438 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 00:58:34.231092 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:34.231067 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe31be1_f392_4843_9ec7_6602fb106b53.slice/crio-5f3fbe1f6da28275b3de659e6fc243272c89f09e04eacfd9f8c1f5199ce10679 WatchSource:0}: Error finding container 5f3fbe1f6da28275b3de659e6fc243272c89f09e04eacfd9f8c1f5199ce10679: Status 404 returned error can't find the container with id 5f3fbe1f6da28275b3de659e6fc243272c89f09e04eacfd9f8c1f5199ce10679 Oct 30 00:58:34.849029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.848992 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" event={"ID":"6128a092-1455-4238-b4d6-57966665f1e3","Type":"ContainerStarted","Data":"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d"} Oct 30 00:58:34.849224 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.849046 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" event={"ID":"6128a092-1455-4238-b4d6-57966665f1e3","Type":"ContainerStarted","Data":"a05dc89ce92447245565e11ccb26d1c4fb8b2aca6303c7d8306d5f159ea54fc3"} Oct 30 00:58:34.849224 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.849097 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:34.850783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.850764 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" event={"ID":"8fe31be1-f392-4843-9ec7-6602fb106b53","Type":"ContainerStarted","Data":"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75"} Oct 30 00:58:34.850904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.850796 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" event={"ID":"8fe31be1-f392-4843-9ec7-6602fb106b53","Type":"ContainerStarted","Data":"5f3fbe1f6da28275b3de659e6fc243272c89f09e04eacfd9f8c1f5199ce10679"} Oct 30 00:58:34.850904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.850821 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:34.889094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.889054 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" podStartSLOduration=1.8890423090000001 podStartE2EDuration="1.889042309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:34.887537321 +0000 UTC m=+1567.127772433" watchObservedRunningTime="2026-10-30 00:58:34.889042309 +0000 UTC m=+1567.129277422" Oct 30 00:58:34.929417 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:34.929374 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" podStartSLOduration=1.929362998 podStartE2EDuration="1.929362998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:34.92546436 +0000 UTC m=+1567.165699484" watchObservedRunningTime="2026-10-30 00:58:34.929362998 +0000 UTC m=+1567.169598111" Oct 30 00:58:35.136063 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:35.136030 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:37.008472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:37.008437 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:37.032163 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:37.032128 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2"] Oct 30 00:58:37.043144 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:37.043113 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6baa951c-j7qr2"] Oct 30 00:58:37.881804 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:37.881768 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d0e260-0cf8-4f20-acfc-b73be720cccd" path="/var/lib/kubelet/pods/84d0e260-0cf8-4f20-acfc-b73be720cccd/volumes" Oct 30 00:58:38.164576 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:38.164482 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:39.205072 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:39.205038 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.080082 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:41.080046 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.232444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:41.232413 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.374126 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:41.374072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.029536 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:43.029507 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.271929 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:43.271883 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:43.744008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:43.743977 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:44.228240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:44.228201 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:44.476436 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:44.476397 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:45.022640 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.022608 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:45.034461 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.034428 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8"] Oct 30 00:58:45.045029 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.045003 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d9aaf92-tdgk8"] Oct 30 00:58:45.210883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.210789 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:45.863457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.863428 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 00:58:45.863962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.863544 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 00:58:45.885362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:45.885333 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be70a5ca-16a8-48ee-987d-45e2c5fe236c" path="/var/lib/kubelet/pods/be70a5ca-16a8-48ee-987d-45e2c5fe236c/volumes" Oct 30 00:58:46.704077 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.704036 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 00:58:46.710225 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.710197 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.758362 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.758167 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 00:58:46.812799 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.812767 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.812989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.812923 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.812989 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.812983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813083 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813009 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813183 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813156 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813381 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813362 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813458 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813629 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813611 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdfl\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.813671 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.813652 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.915724 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.915692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.915782 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.915843 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.915920 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.915971 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916128 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916197 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916201 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916256 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916434 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdfl\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916547 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916639 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.916753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.916678 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.918686 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.918659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.919019 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.918995 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.939975 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.939949 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:46.940162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:46.939974 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdfl\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:47.032106 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.032021 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:47.351981 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.343204 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:47.417541 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.417516 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 00:58:47.420801 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:47.420770 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16ade73_e08d_4fed_baf2_a2d3f11e97a3.slice/crio-ddd8b3f1b26fa66e6d3b70a58ddb8cb633d0a802b3d7251587727727633420b3 WatchSource:0}: Error finding container ddd8b3f1b26fa66e6d3b70a58ddb8cb633d0a802b3d7251587727727633420b3: Status 404 returned error can't find the container with id ddd8b3f1b26fa66e6d3b70a58ddb8cb633d0a802b3d7251587727727633420b3 Oct 30 00:58:47.424129 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.424092 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:47.424240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.424164 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:47.424240 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:47.424197 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:48.020394 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.020341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" event={"ID":"c16ade73-e08d-4fed-baf2-a2d3f11e97a3","Type":"ContainerStarted","Data":"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021"} Oct 30 00:58:48.020947 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.020416 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" event={"ID":"c16ade73-e08d-4fed-baf2-a2d3f11e97a3","Type":"ContainerStarted","Data":"ddd8b3f1b26fa66e6d3b70a58ddb8cb633d0a802b3d7251587727727633420b3"} Oct 30 00:58:48.042229 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.042188 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:48.043760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.043730 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:48.043955 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.043801 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:48.072457 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.072408 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podStartSLOduration=2.072396855 podStartE2EDuration="2.072396855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:48.070296326 +0000 UTC m=+1580.310531463" watchObservedRunningTime="2026-10-30 00:58:48.072396855 +0000 UTC m=+1580.312631967" Oct 30 00:58:48.475608 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.475575 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:48.619280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.619241 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:48.917889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.917720 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:58:48.918052 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:48.917987 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" podUID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" containerName="maas-api" containerID="cri-o://5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de" gracePeriod=30 Oct 30 00:58:49.015158 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.015030 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:49.032603 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.032569 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw"] Oct 30 00:58:49.033069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.032931 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:49.033069 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.032986 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:49.036472 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.036445 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4739c7bc-ptrfw"] Oct 30 00:58:49.481887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.479547 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:58:49.672978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.605996 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\") pod \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " Oct 30 00:58:49.672978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.606122 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") pod \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " Oct 30 00:58:49.672978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.606182 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\") pod \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " Oct 30 00:58:49.672978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.606340 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\" (UniqueName: \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\") pod \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\" (UID: \"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9\") " Oct 30 00:58:49.672978 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.612043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca" pod "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" (UID: "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:58:49.725888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.689334 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd" pod "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" (UID: "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9"). InnerVolumeSpecName "kube-api-access-x8thd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:58:49.725888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.709884 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8thd\" (UniqueName: \"kubernetes.io/projected/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-kube-api-access-x8thd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:49.725888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.709915 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:49.739578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.739538 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls" pod "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" (UID: "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:49.741907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.740555 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs" pod "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" (UID: "6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:58:49.814890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.811758 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:49.814890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.811801 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:58:49.880643 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:49.880605 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946f2cc2-9911-428e-af5d-4e0492b44370" path="/var/lib/kubelet/pods/946f2cc2-9911-428e-af5d-4e0492b44370/volumes" Oct 30 00:58:50.015432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.015385 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:50.033084 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.033056 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:50.033570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.033120 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:50.042473 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.042449 5938 generic.go:356] "Generic (PLEG): container finished" podID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" containerID="5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de" exitCode=0 Oct 30 00:58:50.042616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.042495 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" Oct 30 00:58:50.042616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.042541 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" event={"ID":"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9","Type":"ContainerDied","Data":"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de"} Oct 30 00:58:50.042616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.042593 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx" event={"ID":"6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9","Type":"ContainerDied","Data":"23b6f66c1a81d9b8b32a532f49970bf03363c2b3f374d63b0ea079c00df6965f"} Oct 30 00:58:50.042616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.042616 5938 scope.go:122] "RemoveContainer" containerID="5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de" Oct 30 00:58:50.056792 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.056773 5938 scope.go:122] "RemoveContainer" containerID="5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de" Oct 30 00:58:50.057547 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:50.057528 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de\": container with ID starting with 5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de not found: ID does not exist" containerID="5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de" Oct 30 00:58:50.057626 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.057556 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de"} err="failed to get container status \"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de\": rpc error: code = NotFound desc = could not find container \"5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de\": container with ID starting with 5c1ab2da358e4ddec215b81e531ea3b3b3888e4e09034877755f689df1eb29de not found: ID does not exist" Oct 30 00:58:50.119404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.119376 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:58:50.123130 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.123110 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fa8dd0-5ccbdc96fc-smnnx"] Oct 30 00:58:50.154919 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.154857 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:50.452663 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.452631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:50.520555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.520528 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": read tcp 10.134.0.2:56086->10.134.0.192:15021: read: connection reset by peer" start-of-body= Oct 30 00:58:50.520787 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.520759 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": read tcp 10.134.0.2:56086->10.134.0.192:15021: read: connection reset by peer" Oct 30 00:58:50.672885 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:50.672810 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.032494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.032453 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:51.032715 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.032508 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:51.060249 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.060210 5938 generic.go:356] "Generic (PLEG): container finished" podID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerID="108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005" exitCode=0 Oct 30 00:58:51.060752 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.060407 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerDied","Data":"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005"} Oct 30 00:58:51.060994 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.060969 5938 scope.go:122] "RemoveContainer" containerID="108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005" Oct 30 00:58:51.063520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.063478 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:51.063630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.063579 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:51.063697 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.063630 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:58:51.105756 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.105728 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.221771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.221743 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 00:58:51.222719 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.222700 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" containerName="maas-api" Oct 30 00:58:51.222786 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.222721 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" containerName="maas-api" Oct 30 00:58:51.222846 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.222838 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" containerName="maas-api" Oct 30 00:58:51.225713 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.225694 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.246901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.246351 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b1f6ed-dockercfg-4fq2f\"" Oct 30 00:58:51.250592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.247477 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b1f6ed\"" Oct 30 00:58:51.250592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.248697 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nj7\" (UniqueName: \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.250592 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.249352 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.255588 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.255208 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.257280 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.257253 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 00:58:51.373887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.370681 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nj7\" (UniqueName: \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.373887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.370942 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.450892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.429612 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nj7\" (UniqueName: \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.450892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.443129 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.565892 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.562461 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:51.589149 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.589005 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 00:58:51.683904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.681680 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 00:58:51.683904 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.681799 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.720888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.718493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.720888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.718725 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbm82\" (UniqueName: \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.823104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.823067 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbm82\" (UniqueName: \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.823343 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.823323 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.824510 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.824484 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.847602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.847572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbm82\" (UniqueName: \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\") pod \"payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:51.888451 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:51.888418 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9" path="/var/lib/kubelet/pods/6f2cbe62-d2d8-45cc-94bd-4c9e041a34f9/volumes" Oct 30 00:58:52.016324 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.016286 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 00:58:52.019814 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:58:52.019790 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bc6a7b_9e93_4970_a71b_0abc72258c1e.slice/crio-30b3b259c432f35cc78b15ffc65f8a7a99c806186ae873bbcc79b6bd91209535 WatchSource:0}: Error finding container 30b3b259c432f35cc78b15ffc65f8a7a99c806186ae873bbcc79b6bd91209535: Status 404 returned error can't find the container with id 30b3b259c432f35cc78b15ffc65f8a7a99c806186ae873bbcc79b6bd91209535 Oct 30 00:58:52.032321 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.032298 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:52.032419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.032346 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:52.064117 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.064090 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:52.073412 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.073384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" event={"ID":"38bc6a7b-9e93-4970-a71b-0abc72258c1e","Type":"ContainerStarted","Data":"30b3b259c432f35cc78b15ffc65f8a7a99c806186ae873bbcc79b6bd91209535"} Oct 30 00:58:52.075939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.075914 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerStarted","Data":"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf"} Oct 30 00:58:52.443951 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.442765 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 00:58:52.572299 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.570263 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:52.762226 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.762148 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:58:52.762432 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.762304 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:52.762555 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:52.762539 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:53.034657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.034560 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 00:58:53.034657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.034632 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 00:58:53.155011 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.154980 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" event={"ID":"4d36a46e-35e7-4ef9-9124-be9f76886052","Type":"ContainerStarted","Data":"750d41af1d8cc63e0b87918a0456c65cd74a3e8e4e83e24ef1be50accbe5eff6"} Oct 30 00:58:53.155460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.155441 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" event={"ID":"4d36a46e-35e7-4ef9-9124-be9f76886052","Type":"ContainerStarted","Data":"6e7ed20c816bcb45fb4aaf03475ff3d57446f21c54bc89ef7008aa380135924d"} Oct 30 00:58:53.156162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.156138 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:58:53.158474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.158453 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" event={"ID":"38bc6a7b-9e93-4970-a71b-0abc72258c1e","Type":"ContainerStarted","Data":"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e"} Oct 30 00:58:53.158605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.158595 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:58:53.241267 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.241209 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" podStartSLOduration=2.241193567 podStartE2EDuration="2.241193567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:53.197239473 +0000 UTC m=+1585.437474588" watchObservedRunningTime="2026-10-30 00:58:53.241193567 +0000 UTC m=+1585.481428683" Oct 30 00:58:53.252658 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.252608 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" podStartSLOduration=2.25259463 podStartE2EDuration="2.25259463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:53.248271387 +0000 UTC m=+1585.488506501" watchObservedRunningTime="2026-10-30 00:58:53.25259463 +0000 UTC m=+1585.492829743" Oct 30 00:58:53.762357 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.762302 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:53.762556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:53.762397 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:54.075352 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.075270 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:54.076767 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.076745 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:54.078800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.078772 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 00:58:54.471997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.471962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:54.762006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.761928 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:54.762006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:54.761977 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:55.761772 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:55.761741 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:55.762243 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:55.761789 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:56.236409 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:56.236362 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:56.762033 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:56.762003 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:56.762429 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:56.762053 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:57.628587 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.628546 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:57.764887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.763249 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 00:58:57.764887 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.763326 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 00:58:57.781888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.781778 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:57.963134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.963034 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:58:57.963328 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:57.963235 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" containerID="cri-o://bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6" gracePeriod=30 Oct 30 00:58:58.029453 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:58.025283 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:58.420678 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:58.418423 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:58.767993 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:58.767919 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:58:58.768468 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:58.768447 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:58:58.769185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:58:58.769165 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 00:58:58.837186 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:58:58.837156 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:59:00.344601 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:00.337887 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.162717 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:02.162682 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.258820 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:02.258784 5938 patch_prober.go:29] interesting pod/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 00:59:02.259076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:02.258851 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 00:59:02.315105 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:02.315076 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:02.764800 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:02.764721 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:03.171368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.171339 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 00:59:03.179215 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.179185 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.202987 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.202854 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5007e1-openshift-default-dockercfg-h59nd\"" Oct 30 00:59:03.210516 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.210491 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 00:59:03.256107 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.256072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:03.272556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.272533 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.414553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.414716 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.414765 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.414799 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.414899 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415040 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415160 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415254 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415316 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\") pod \"c9308ee3-b32d-4860-ba78-9c943a245daf\" (UID: \"c9308ee3-b32d-4860-ba78-9c943a245daf\") " Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415554 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415587 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.415699 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7j7p\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.428171 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.432937 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.435362 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:03.506578 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.487631 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531234 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531585 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531669 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531697 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531915 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.531922 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531935 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.532504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531949 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.532504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.531964 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.549659 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.535464 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:03.554307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.550299 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:03.554307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.553784 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:03.556939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.556844 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "kube-api-access-8qpjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:03.559025 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.558731 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:59:03.561582 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.561540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token" pod "c9308ee3-b32d-4860-ba78-9c943a245daf" (UID: "c9308ee3-b32d-4860-ba78-9c943a245daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:03.574195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.572647 5938 generic.go:356] "Generic (PLEG): container finished" podID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerID="bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6" exitCode=0 Oct 30 00:59:03.574195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.572844 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerDied","Data":"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6"} Oct 30 00:59:03.574195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.572935 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" event={"ID":"c9308ee3-b32d-4860-ba78-9c943a245daf","Type":"ContainerDied","Data":"bc6a243b10fd4ac8095a4858cf159404b512826d7dc77cbda3ae3510d1a2f31b"} Oct 30 00:59:03.574195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.572960 5938 scope.go:122] "RemoveContainer" containerID="bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6" Oct 30 00:59:03.574195 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.573112 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc" Oct 30 00:59:03.590969 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.590948 5938 scope.go:122] "RemoveContainer" containerID="e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10" Oct 30 00:59:03.603406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.603384 5938 scope.go:122] "RemoveContainer" containerID="bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6" Oct 30 00:59:03.603719 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:03.603699 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6\": container with ID starting with bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6 not found: ID does not exist" containerID="bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6" Oct 30 00:59:03.603801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.603727 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6"} err="failed to get container status \"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6\": rpc error: code = NotFound desc = could not find container \"bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6\": container with ID starting with bf6ee878dedbdbc2d29a4b9536efc10f06d0fbd4251fea705279f68c51cfd4d6 not found: ID does not exist" Oct 30 00:59:03.603801 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.603746 5938 scope.go:122] "RemoveContainer" containerID="e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10" Oct 30 00:59:03.604053 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:03.604029 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10\": container with ID starting with e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10 not found: ID does not exist" containerID="e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10" Oct 30 00:59:03.604096 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.604063 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10"} err="failed to get container status \"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10\": rpc error: code = NotFound desc = could not find container \"e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10\": container with ID starting with e95197d474bd1292ec9216e4f07d0fa1916775dfd436d0a7d54e65c0c1314c10 not found: ID does not exist" Oct 30 00:59:03.631508 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.631477 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:59:03.633364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633290 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.633526 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633431 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.633615 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633562 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.633672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633648 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.633740 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633676 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633752 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633780 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.633893 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7j7p\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634037 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634134 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634150 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpjk\" (UniqueName: \"kubernetes.io/projected/c9308ee3-b32d-4860-ba78-9c943a245daf-kube-api-access-8qpjk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634166 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634180 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9308ee3-b32d-4860-ba78-9c943a245daf-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634193 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9308ee3-b32d-4860-ba78-9c943a245daf-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634350 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634392 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634922 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.634978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636001 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.635206 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.636811 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.636396 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fa8dd0-openshift-default-6588f97d75-8dzwc"] Oct 30 00:59:03.637260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.637234 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.637392 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.637348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.656976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.656947 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.657078 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.657012 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7j7p\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\") pod \"e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.801364 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.801287 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:03.885376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:03.884883 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" path="/var/lib/kubelet/pods/c9308ee3-b32d-4860-ba78-9c943a245daf/volumes" Oct 30 00:59:04.005764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.005732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:04.122135 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.122107 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 00:59:04.123898 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:59:04.123858 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f03d8c_6c8b_411f_a279_71375d4597c9.slice/crio-8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f WatchSource:0}: Error finding container 8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f: Status 404 returned error can't find the container with id 8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f Oct 30 00:59:04.127148 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.127106 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:04.127238 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.127196 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:04.127279 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.127241 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:04.169197 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.169179 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 00:59:04.596939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.596811 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerStarted","Data":"9c85deb20f58c7160245b6feebb9ae3b51daf9d611aaabda3a7a7cb82ecd120e"} Oct 30 00:59:04.596939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.596916 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerStarted","Data":"8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f"} Oct 30 00:59:04.642627 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.642579 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podStartSLOduration=1.6425660149999999 podStartE2EDuration="1.642566015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:04.639267041 +0000 UTC m=+1596.879502155" watchObservedRunningTime="2026-10-30 00:59:04.642566015 +0000 UTC m=+1596.882801128" Oct 30 00:59:04.801957 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:04.801922 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:05.176104 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:05.176078 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 00:59:05.807999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:05.807970 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:05.821006 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:05.820978 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:06.614966 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:06.614933 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:06.615755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:06.615737 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 00:59:07.149675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:07.149639 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:07.304181 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:07.304144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:08.091935 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:08.091898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:09.143901 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.134720 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:09.273217 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.273112 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.394243 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.395718 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.395744 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.395801 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.395809 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:09.396941 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.396228 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:09.488368 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.488337 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.550894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.541259 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5007e1\"" Oct 30 00:59:09.550894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.541360 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5007e1-dockercfg-wdbz8\"" Oct 30 00:59:09.605909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.603748 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.605909 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.603893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668m9\" (UniqueName: \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.649943 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.629736 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 00:59:09.710070 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.710034 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668m9\" (UniqueName: \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.710605 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.710577 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.711518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.711500 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.732821 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.732799 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668m9\" (UniqueName: \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\") pod \"payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:09.930542 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:09.930466 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:10.267206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.267181 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 00:59:10.268653 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:59:10.268625 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00ea3f2_557c_41d3_9a21_5ffa70b65b0f.slice/crio-8006d64156b1356224a360078be0044ea6246774a161533806956be23d26a93e WatchSource:0}: Error finding container 8006d64156b1356224a360078be0044ea6246774a161533806956be23d26a93e: Status 404 returned error can't find the container with id 8006d64156b1356224a360078be0044ea6246774a161533806956be23d26a93e Oct 30 00:59:10.436311 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.436281 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:59:10.469895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.469847 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:59:10.564570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.564481 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:10.705891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.694271 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" event={"ID":"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f","Type":"ContainerStarted","Data":"464046499d8d1d34ec0e8883b506bdbe21a61331e6fface0594ce60b6b2b66f3"} Oct 30 00:59:10.705891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.694335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" event={"ID":"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f","Type":"ContainerStarted","Data":"8006d64156b1356224a360078be0044ea6246774a161533806956be23d26a93e"} Oct 30 00:59:10.705891 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.695001 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:10.728272 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.728230 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:10.743363 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:10.743297 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" podStartSLOduration=1.743283036 podStartE2EDuration="1.743283036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:10.738174373 +0000 UTC m=+1602.978409489" watchObservedRunningTime="2026-10-30 00:59:10.743283036 +0000 UTC m=+1602.983518149" Oct 30 00:59:12.373327 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:12.373293 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:14.111895 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:14.110544 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:15.437474 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.437435 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" podUID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" containerName="payload-processing" containerID="cri-o://ca03814a44410ef772c7df306ed491b2804a5f0e7496681dfe02336224066bda" gracePeriod=25 Oct 30 00:59:15.470985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.470944 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" podUID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" containerName="payload-pre-processing" containerID="cri-o://a93f845aa0a4eeb37b29575c41e9362d84ad9567c99957200d967527682a3a39" gracePeriod=25 Oct 30 00:59:15.735736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.733890 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 00:59:15.735736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.735484 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9308ee3-b32d-4860-ba78-9c943a245daf" containerName="istio-proxy" Oct 30 00:59:15.740376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.740335 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.754783 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.753993 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 00:59:15.765415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.763597 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-cce513-serving-cert\"" Oct 30 00:59:15.775334 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.775125 5938 generic.go:356] "Generic (PLEG): container finished" podID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" containerID="ca03814a44410ef772c7df306ed491b2804a5f0e7496681dfe02336224066bda" exitCode=0 Oct 30 00:59:15.775494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.775370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" event={"ID":"f6eb19f7-f78e-41f7-b1c4-4b3948116b29","Type":"ContainerDied","Data":"ca03814a44410ef772c7df306ed491b2804a5f0e7496681dfe02336224066bda"} Oct 30 00:59:15.775494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.775422 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" event={"ID":"f6eb19f7-f78e-41f7-b1c4-4b3948116b29","Type":"ContainerDied","Data":"8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba"} Oct 30 00:59:15.775494 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.775450 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8534299ea09eb4de4beaac8fdb15e1475f9a1ee23654a686c50a7032365e4aba" Oct 30 00:59:15.780967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.780003 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.780967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.780047 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.780967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.780118 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.780967 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.780156 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tl9f\" (UniqueName: \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.789230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.782847 5938 generic.go:356] "Generic (PLEG): container finished" podID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" containerID="a93f845aa0a4eeb37b29575c41e9362d84ad9567c99957200d967527682a3a39" exitCode=0 Oct 30 00:59:15.789230 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.782977 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" event={"ID":"d378156c-7b5b-4b2c-b2bb-0daedbc3908b","Type":"ContainerDied","Data":"a93f845aa0a4eeb37b29575c41e9362d84ad9567c99957200d967527682a3a39"} Oct 30 00:59:15.830708 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.830680 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:59:15.855303 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.855278 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:15.876275 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.876251 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:59:15.882630 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.882591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\" (UniqueName: \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\") pod \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " Oct 30 00:59:15.882849 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.882834 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\") pod \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\" (UID: \"f6eb19f7-f78e-41f7-b1c4-4b3948116b29\") " Oct 30 00:59:15.883193 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.883173 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tl9f\" (UniqueName: \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.883518 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.883491 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.883602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.883526 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.883602 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.883597 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.884155 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.883796 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume" pod "f6eb19f7-f78e-41f7-b1c4-4b3948116b29" (UID: "f6eb19f7-f78e-41f7-b1c4-4b3948116b29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:15.884570 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.884545 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.889444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.888528 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.889444 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.888641 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.889974 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.889949 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8" pod "f6eb19f7-f78e-41f7-b1c4-4b3948116b29" (UID: "f6eb19f7-f78e-41f7-b1c4-4b3948116b29"). InnerVolumeSpecName "kube-api-access-dh2b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:15.906657 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.906628 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tl9f\" (UniqueName: \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\") pod \"maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:15.993976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.988621 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\" (UniqueName: \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\") pod \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " Oct 30 00:59:15.993976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.989063 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\") pod \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\" (UID: \"d378156c-7b5b-4b2c-b2bb-0daedbc3908b\") " Oct 30 00:59:15.993976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.992107 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume" pod "d378156c-7b5b-4b2c-b2bb-0daedbc3908b" (UID: "d378156c-7b5b-4b2c-b2bb-0daedbc3908b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:15.993976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.993824 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt" pod "d378156c-7b5b-4b2c-b2bb-0daedbc3908b" (UID: "d378156c-7b5b-4b2c-b2bb-0daedbc3908b"). InnerVolumeSpecName "kube-api-access-7qbkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:15.994410 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.994387 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbkt\" (UniqueName: \"kubernetes.io/projected/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-kube-api-access-7qbkt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:15.994505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.994413 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:15.994505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.994426 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2b8\" (UniqueName: \"kubernetes.io/projected/f6eb19f7-f78e-41f7-b1c4-4b3948116b29-kube-api-access-dh2b8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:15.994505 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:15.994440 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d378156c-7b5b-4b2c-b2bb-0daedbc3908b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:16.004750 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.004631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:16.120707 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.120670 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:16.434753 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.434723 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 00:59:16.435355 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:59:16.435330 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0aaf4e_f905_4fd1_9276_1389c2261810.slice/crio-6be37547e611bf6cf36c8226f46bb9d30193ee56292d3690302b35d8f979f237 WatchSource:0}: Error finding container 6be37547e611bf6cf36c8226f46bb9d30193ee56292d3690302b35d8f979f237: Status 404 returned error can't find the container with id 6be37547e611bf6cf36c8226f46bb9d30193ee56292d3690302b35d8f979f237 Oct 30 00:59:16.437079 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.437059 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:59:16.792416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.792337 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" Oct 30 00:59:16.792416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.792377 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc" event={"ID":"d378156c-7b5b-4b2c-b2bb-0daedbc3908b","Type":"ContainerDied","Data":"f143192fdc47959939a95e8bde3de3a1f2d25336e47950f649609a0834844919"} Oct 30 00:59:16.793039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.792443 5938 scope.go:122] "RemoveContainer" containerID="a93f845aa0a4eeb37b29575c41e9362d84ad9567c99957200d967527682a3a39" Oct 30 00:59:16.794030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.793919 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" event={"ID":"ea0aaf4e-f905-4fd1-9276-1389c2261810","Type":"ContainerStarted","Data":"6be37547e611bf6cf36c8226f46bb9d30193ee56292d3690302b35d8f979f237"} Oct 30 00:59:16.794030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.793943 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5" Oct 30 00:59:16.845976 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.845954 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:59:16.851282 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.851260 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bab19a-769d9d858c-gq8fc"] Oct 30 00:59:16.880159 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.880104 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:59:16.884894 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:16.884872 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bab19a-665c67756-zz5k5"] Oct 30 00:59:17.125888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.124059 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:59:17.125888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.124121 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:17.760134 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.760094 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:17.810556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.810514 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" event={"ID":"ea0aaf4e-f905-4fd1-9276-1389c2261810","Type":"ContainerStarted","Data":"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9"} Oct 30 00:59:17.811032 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.810598 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:17.850999 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.850934 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" podStartSLOduration=2.470908579 podStartE2EDuration="2.850915126s" totalImagesPullingTime="380.006547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:17.84967873 +0000 UTC m=+1610.089913844" watchObservedRunningTime="2026-10-30 00:59:17.850915126 +0000 UTC m=+1610.091150242" Oct 30 00:59:17.881094 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.881062 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" path="/var/lib/kubelet/pods/d378156c-7b5b-4b2c-b2bb-0daedbc3908b/volumes" Oct 30 00:59:17.881556 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:17.881541 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" path="/var/lib/kubelet/pods/f6eb19f7-f78e-41f7-b1c4-4b3948116b29/volumes" Oct 30 00:59:18.823853 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.823760 5938 generic.go:356] "Generic (PLEG): container finished" podID="d7b0d383-83ea-444e-903f-d50785175255" containerID="a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db" exitCode=0 Oct 30 00:59:18.824317 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.823845 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerDied","Data":"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db"} Oct 30 00:59:18.824459 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.824443 5938 scope.go:122] "RemoveContainer" containerID="a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db" Oct 30 00:59:18.826255 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.826223 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:18.826390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.826298 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:18.826390 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:18.826331 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 00:59:19.544883 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:19.544533 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:19.834162 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:19.834083 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerStarted","Data":"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd"} Oct 30 00:59:20.758760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:20.758728 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:59:20.759415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:20.758949 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 00:59:20.759415 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:20.758996 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 00:59:21.478898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:21.475963 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:21.718714 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:21.718687 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 00:59:21.758760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:21.758687 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 00:59:21.758760 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:21.758737 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 00:59:22.758771 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:22.758737 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 00:59:22.759185 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:22.758791 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 00:59:23.105890 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:23.104029 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:23.758934 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:23.758903 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 00:59:23.759333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:23.758955 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 00:59:23.829896 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:23.829849 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 00:59:24.763889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:24.759652 5938 patch_prober.go:29] interesting pod/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 00:59:24.763889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:24.759733 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 00:59:24.766676 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:24.766649 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:25.762792 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:25.762764 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:59:25.763146 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:25.763127 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:59:25.763819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:25.763800 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 00:59:26.622206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:26.622164 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:28.414333 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:28.414097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:28.836948 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:28.836832 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:59:29.838020 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:29.837986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:30.306389 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:30.306336 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:30.635376 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:30.635337 5938 scope.go:122] "RemoveContainer" containerID="5a92af42834fddb86c8ef5216d7a874e2fe46d0691113ff9d7723533aab549e4" Oct 30 00:59:30.646460 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:30.646441 5938 scope.go:122] "RemoveContainer" containerID="216f1ff5c6e9ffe3f1994c6f713efb240df02dcc2729418d9daf1f92a4a29530" Oct 30 00:59:31.522003 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:31.521964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.228262 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.228232 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 00:59:32.229119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229095 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" containerName="payload-pre-processing" Oct 30 00:59:32.229119 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229118 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" containerName="payload-pre-processing" Oct 30 00:59:32.229306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229154 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" containerName="payload-processing" Oct 30 00:59:32.229306 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229162 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" containerName="payload-processing" Oct 30 00:59:32.229416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229384 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d378156c-7b5b-4b2c-b2bb-0daedbc3908b" containerName="payload-pre-processing" Oct 30 00:59:32.229416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.229407 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6eb19f7-f78e-41f7-b1c4-4b3948116b29" containerName="payload-processing" Oct 30 00:59:32.235030 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.235008 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.250739 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.250717 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 00:59:32.256988 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.256964 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b1f6ed-serving-cert\"" Oct 30 00:59:32.294903 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.294833 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwbd\" (UniqueName: \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.295066 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.294914 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.295204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.295183 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.295404 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.295353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.352655 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.352631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.396808 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.396776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.396985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.396831 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwbd\" (UniqueName: \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.396985 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.396890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.397095 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:32.396988 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-serving-cert: secret "maas-api-e2e-inf-b-b1f6ed-serving-cert" not found Oct 30 00:59:32.397095 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:32.397043 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls podName:7170a77a-f354-4f2f-8c6d-b16ef36eea08 nodeName:}" failed. No retries permitted until 2026-10-30 00:59:32.897030152 +0000 UTC m=+1625.137265244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls") pod "maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" (UID: "7170a77a-f354-4f2f-8c6d-b16ef36eea08") : secret "maas-api-e2e-inf-b-b1f6ed-serving-cert" not found Oct 30 00:59:32.397194 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.397161 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.397764 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.397739 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.399342 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.399322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.424277 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.424221 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwbd\" (UniqueName: \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.500206 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.500090 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.768819 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.768715 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:32.904538 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.904505 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:32.907569 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:32.907531 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:33.154888 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:33.152370 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:33.493076 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:33.493047 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 00:59:33.493643 ip-10-0-131-130 kubenswrapper[5938]: W1030 00:59:33.493619 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7170a77a_f354_4f2f_8c6d_b16ef36eea08.slice/crio-e464f55aeadd47c88953aca3ee448d7f0418da043e5ce4d1f764be8207275e3e WatchSource:0}: Error finding container e464f55aeadd47c88953aca3ee448d7f0418da043e5ce4d1f764be8207275e3e: Status 404 returned error can't find the container with id e464f55aeadd47c88953aca3ee448d7f0418da043e5ce4d1f764be8207275e3e Oct 30 00:59:33.979907 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:33.979847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" event={"ID":"7170a77a-f354-4f2f-8c6d-b16ef36eea08","Type":"ContainerStarted","Data":"e464f55aeadd47c88953aca3ee448d7f0418da043e5ce4d1f764be8207275e3e"} Oct 30 00:59:34.392204 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:34.392172 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:34.989618 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:34.989583 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" event={"ID":"7170a77a-f354-4f2f-8c6d-b16ef36eea08","Type":"ContainerStarted","Data":"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d"} Oct 30 00:59:34.990087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:34.989672 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:35.030736 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:35.030695 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" podStartSLOduration=2.61303854 podStartE2EDuration="3.030683379s" totalImagesPullingTime="417.644839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:35.029335857 +0000 UTC m=+1627.269570972" watchObservedRunningTime="2026-10-30 00:59:35.030683379 +0000 UTC m=+1627.270918491" Oct 30 00:59:36.074520 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:36.074483 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:36.999844 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:36.999790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:37.310666 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:37.310588 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:59:37.311202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:37.310828 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" containerID="cri-o://d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82" gracePeriod=30 Oct 30 00:59:37.998818 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:37.998785 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:38.363136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:38.363095 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:39.691008 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:39.690759 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:41.002889 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:41.002846 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 00:59:41.323654 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:41.323569 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:42.606742 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.606719 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:59:42.630087 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630061 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630202 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630177 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630260 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630237 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630348 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630328 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:42.630419 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630333 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630478 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630553 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630521 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:42.630616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630546 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:42.630616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630562 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630616 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630593 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630678 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630742 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\") pod \"2181c326-e2df-45bc-b795-9049cafad0fe\" (UID: \"2181c326-e2df-45bc-b795-9049cafad0fe\") " Oct 30 00:59:42.630832 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.630789 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:42.631065 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631033 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:59:42.631645 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631620 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.631755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631651 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.631755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631667 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.631755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631683 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.631755 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.631699 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2181c326-e2df-45bc-b795-9049cafad0fe-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.632672 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.632642 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 00:59:42.633039 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.633017 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:59:42.633136 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.633065 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "kube-api-access-4tbbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:42.633192 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.633144 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token" pod "2181c326-e2df-45bc-b795-9049cafad0fe" (UID: "2181c326-e2df-45bc-b795-9049cafad0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:59:42.732952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.732917 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.732952 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.732950 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbbw\" (UniqueName: \"kubernetes.io/projected/2181c326-e2df-45bc-b795-9049cafad0fe-kube-api-access-4tbbw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.733169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.732962 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2181c326-e2df-45bc-b795-9049cafad0fe-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:42.733169 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:42.732970 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2181c326-e2df-45bc-b795-9049cafad0fe-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 00:59:43.085207 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.085167 5938 generic.go:356] "Generic (PLEG): container finished" podID="2181c326-e2df-45bc-b795-9049cafad0fe" containerID="d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82" exitCode=0 Oct 30 00:59:43.085434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.085221 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" Oct 30 00:59:43.085434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.085239 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" event={"ID":"2181c326-e2df-45bc-b795-9049cafad0fe","Type":"ContainerDied","Data":"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82"} Oct 30 00:59:43.085434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.085295 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp" event={"ID":"2181c326-e2df-45bc-b795-9049cafad0fe","Type":"ContainerDied","Data":"e22657ac981b7754df32e197dab01cf5450e92a5a3f07968d69b4fb459225f15"} Oct 30 00:59:43.085434 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.085328 5938 scope.go:122] "RemoveContainer" containerID="d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82" Oct 30 00:59:43.098319 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.098296 5938 scope.go:122] "RemoveContainer" containerID="d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82" Oct 30 00:59:43.098615 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:43.098597 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82\": container with ID starting with d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82 not found: ID does not exist" containerID="d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82" Oct 30 00:59:43.098677 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.098625 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82"} err="failed to get container status \"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82\": rpc error: code = NotFound desc = could not find container \"d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82\": container with ID starting with d6cedeae20ce52ca155a9e2facc9244196326e1adcec0cdf9de3b5c4dec1fd82 not found: ID does not exist" Oct 30 00:59:43.145675 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.145637 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:59:43.153406 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.153380 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tlpfp"] Oct 30 00:59:43.181301 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.181271 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:43.879416 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:43.879382 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" path="/var/lib/kubelet/pods/2181c326-e2df-45bc-b795-9049cafad0fe/volumes" Oct 30 00:59:44.963851 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:44.963812 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:46.650504 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:46.650470 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:48.532341 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:48.532310 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:48.674560 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:48.674521 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:48.810898 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:48.810779 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:50.380307 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:50.380274 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:52.105962 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:52.105920 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:53.953939 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:53.953900 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:55.649777 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:55.649739 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:57.536371 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:57.536323 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:58.838629 ip-10-0-131-130 kubenswrapper[5938]: E1030 00:59:58.838591 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 00:59:59.028594 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:59.028554 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp"] Oct 30 00:59:59.039384 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:59.039361 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c72ea9b2-5snbp"] Oct 30 00:59:59.364997 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:59.364961 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:59.879012 ip-10-0-131-130 kubenswrapper[5938]: I1030 00:59:59.878984 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832a28fd-9a64-43d7-8d6a-97b4ad5c2946" path="/var/lib/kubelet/pods/832a28fd-9a64-43d7-8d6a-97b4ad5c2946/volumes" Oct 30 01:00:00.172670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.172576 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:00.173441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.173420 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" Oct 30 01:00:00.173441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.173438 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" Oct 30 01:00:00.173648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.173628 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2181c326-e2df-45bc-b795-9049cafad0fe" containerName="istio-proxy" Oct 30 01:00:00.179064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.179045 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.179311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.179285 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:00.214960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.212108 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:00:00.216070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.216045 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:00.217725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.217124 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.222346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.222309 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.233543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.233520 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:00.234808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.234772 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:00.246762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.245287 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:00.248621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.248591 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2c4w\" (UniqueName: \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.253017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.252994 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.253285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.253265 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.255125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.255103 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfwv\" (UniqueName: \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.255335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.255308 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.255609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.255591 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cvv\" (UniqueName: \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.271992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.271967 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:00.275600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.275584 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.281025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.281007 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.349342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.349251 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:00.366369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.366331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.367645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.367614 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfwv\" (UniqueName: \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.368083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.368047 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.368402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.366999 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:00.375723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.368252 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnjf\" (UniqueName: \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.375723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.375183 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.375723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.375390 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cvv\" (UniqueName: \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.375723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.375584 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2c4w\" (UniqueName: \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.377059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.377037 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.377220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.377202 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.377408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.377391 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8dg\" (UniqueName: \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.377546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.377532 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.378362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.378341 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.380068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.380049 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.381719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.381695 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:00.381918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.381804 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.411159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.411135 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:00.411354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.411340 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:00.411495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.411483 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.415952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.415926 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfwv\" (UniqueName: \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.417716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.417694 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2c4w\" (UniqueName: \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\") pod \"maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.419876 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.419848 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:00.420793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.420751 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cvv\" (UniqueName: \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\") pod \"maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.426490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.426449 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.431503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.431485 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:00.438021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.438003 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:00.443479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.443461 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:00.451638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.451067 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:00.453035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.453018 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.458607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.458586 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:00.458818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.458626 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:00.458818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.458651 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:00.458818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.458710 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.479201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.478983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.479201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479096 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.479779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479321 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8dg\" (UniqueName: \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.479779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479697 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzz4g\" (UniqueName: \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.479961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479800 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8qt\" (UniqueName: \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.479961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479913 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx59\" (UniqueName: \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.480078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.479964 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.480078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.480066 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.480200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.480167 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.480260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.480210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnjf\" (UniqueName: \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.480797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.480771 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.480931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.480785 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.514258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.514230 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8dg\" (UniqueName: \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\") pod \"maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.514712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.514687 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnjf\" (UniqueName: \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\") pod \"maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.535669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.535639 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:00.571317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.571288 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:00.581996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.581954 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.582383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.582359 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.614102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.614081 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.614774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.614757 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.614969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.614016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.615032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615017 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cwm\" (UniqueName: \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.615107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615094 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzz4g\" (UniqueName: \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.615144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615125 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8qt\" (UniqueName: \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.615181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615149 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.615181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615173 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx59\" (UniqueName: \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.615254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615213 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.615304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.615292 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh4j\" (UniqueName: \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.616620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.616406 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.656912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.652476 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx59\" (UniqueName: \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\") pod \"maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.656912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.652476 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8qt\" (UniqueName: \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\") pod \"maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.656912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.656844 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzz4g\" (UniqueName: \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\") pod \"maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.672651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.672270 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:00.714996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.714130 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:00.721100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.718383 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cwm\" (UniqueName: \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.721100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.718492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.721100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.718620 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh4j\" (UniqueName: \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.721100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.718694 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.721100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.720006 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.723105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.721675 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:00.723105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.722567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.745001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.744939 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cwm\" (UniqueName: \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\") pod \"maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.746249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.746229 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh4j\" (UniqueName: \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:00.765413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.765313 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:00.781847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.780600 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:00.893279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.827293 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:00.893279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.863649 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:00.960005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:00.946148 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:01.089886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.080774 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:01.169691 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:00:01.169653 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72523efe_c57d_43c5_b44a_a3aa7edea5d7.slice/crio-382c8f3e01310debed31a5b69a9fe7735411bb5e902e659d9c980867dc0afb68 WatchSource:0}: Error finding container 382c8f3e01310debed31a5b69a9fe7735411bb5e902e659d9c980867dc0afb68: Status 404 returned error can't find the container with id 382c8f3e01310debed31a5b69a9fe7735411bb5e902e659d9c980867dc0afb68 Oct 30 01:00:01.181641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.181612 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:01.317586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.293933 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:01.383771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.383732 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:01.400993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.400957 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerStarted","Data":"382c8f3e01310debed31a5b69a9fe7735411bb5e902e659d9c980867dc0afb68"} Oct 30 01:00:01.402599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.402567 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerStarted","Data":"8334fd9623297d27c0e20ef00f96ac7bc73eb61d9cd05cf3e0a977ade5ed1f2b"} Oct 30 01:00:01.562223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.562149 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:01.565201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.565174 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:01.567010 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:00:01.566977 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda88969c_2f54_4e2e_be1f_9c2491b432c7.slice/crio-cb255bdca9e53be30409436c3e83193eefecd293001bbf93991042fc94eecf8a WatchSource:0}: Error finding container cb255bdca9e53be30409436c3e83193eefecd293001bbf93991042fc94eecf8a: Status 404 returned error can't find the container with id cb255bdca9e53be30409436c3e83193eefecd293001bbf93991042fc94eecf8a Oct 30 01:00:01.730785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.730752 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:01.749363 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:00:01.749332 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37221d5b_9f20_4a94_bbc3_6e5837f058c9.slice/crio-099467fcf9bb5200659c553a7bd2e28631abd124f0343ebab1927166c2f1d255 WatchSource:0}: Error finding container 099467fcf9bb5200659c553a7bd2e28631abd124f0343ebab1927166c2f1d255: Status 404 returned error can't find the container with id 099467fcf9bb5200659c553a7bd2e28631abd124f0343ebab1927166c2f1d255 Oct 30 01:00:01.966228 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:00:01.962964 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641aa109_dc7d_4d4e_ba48_a186b6d49a43.slice/crio-918124ef7f969acea80b00ad5c185c5214acc371549b9941de34da70bc087e39 WatchSource:0}: Error finding container 918124ef7f969acea80b00ad5c185c5214acc371549b9941de34da70bc087e39: Status 404 returned error can't find the container with id 918124ef7f969acea80b00ad5c185c5214acc371549b9941de34da70bc087e39 Oct 30 01:00:01.968706 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:00:01.967978 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c87f67e_8633_4511_954e_9d38b1119b97.slice/crio-d8dc3fab6463a5fed395974faee9dd952e6741334db23c40f28b230877a1bea6 WatchSource:0}: Error finding container d8dc3fab6463a5fed395974faee9dd952e6741334db23c40f28b230877a1bea6: Status 404 returned error can't find the container with id d8dc3fab6463a5fed395974faee9dd952e6741334db23c40f28b230877a1bea6 Oct 30 01:00:01.972211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.972192 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:01.972324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.972255 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:01.972324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:01.972286 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:02.016819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.016786 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:02.116808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.116777 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:02.415524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.415497 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerStarted","Data":"6c29ea3c1f344dcb55f071398fbc86a579a5d861bcf23d020aa5b5550950c45f"} Oct 30 01:00:02.416815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.416791 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerStarted","Data":"b994a9518e54b4f6963780100d0d9a056202056e485e4a0a700bff07ee631f38"} Oct 30 01:00:02.418087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.418066 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerStarted","Data":"099467fcf9bb5200659c553a7bd2e28631abd124f0343ebab1927166c2f1d255"} Oct 30 01:00:02.419570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.419548 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerStarted","Data":"eb1cf012a4b341c013287af66f974e62f21da4a89c1cad5816c3ddf4f0ff40fa"} Oct 30 01:00:02.421002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.420980 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerStarted","Data":"918124ef7f969acea80b00ad5c185c5214acc371549b9941de34da70bc087e39"} Oct 30 01:00:02.422662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.422638 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerStarted","Data":"7c08e7a59f7106f246b3c04657cdd29577ba63b12cc98ef662fc1274fc4b9a6a"} Oct 30 01:00:02.422768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.422685 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerStarted","Data":"24c28a5339b67c02bc842e00a9a937ed5ba478562bd732f6298ffac7701c9073"} Oct 30 01:00:02.423938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.423910 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerStarted","Data":"d8dc3fab6463a5fed395974faee9dd952e6741334db23c40f28b230877a1bea6"} Oct 30 01:00:02.425474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.425452 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerStarted","Data":"fda15610e9a4704784d9916e2403379aea06eb48d2e4515607e4b1aa8f9ebdf4"} Oct 30 01:00:02.427201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.427177 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerStarted","Data":"b02d6ed4e01f8dc15e1bf6b58b7a5af82702faf58ce457299e760bf2298c30ea"} Oct 30 01:00:02.427289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.427218 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerStarted","Data":"cb255bdca9e53be30409436c3e83193eefecd293001bbf93991042fc94eecf8a"} Oct 30 01:00:02.428746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.428727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerStarted","Data":"85d5d55507cc4eafb7a0362a198cda8dac1b556d636caf6d829191c43deaa62f"} Oct 30 01:00:02.465274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.465202 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" podStartSLOduration=2.465190883 podStartE2EDuration="2.465190883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.460154048 +0000 UTC m=+1654.700389164" watchObservedRunningTime="2026-10-30 01:00:02.465190883 +0000 UTC m=+1654.705425989" Oct 30 01:00:02.497558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.497511 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" podStartSLOduration=2.497496927 podStartE2EDuration="2.497496927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.497252177 +0000 UTC m=+1654.737487290" watchObservedRunningTime="2026-10-30 01:00:02.497496927 +0000 UTC m=+1654.737732043" Oct 30 01:00:02.535905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.535834 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" podStartSLOduration=2.5358154280000003 podStartE2EDuration="2.535815428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.534521174 +0000 UTC m=+1654.774756287" watchObservedRunningTime="2026-10-30 01:00:02.535815428 +0000 UTC m=+1654.776050564" Oct 30 01:00:02.572486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:02.572445 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" podStartSLOduration=2.572433007 podStartE2EDuration="2.572433007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:02.571712523 +0000 UTC m=+1654.811947638" watchObservedRunningTime="2026-10-30 01:00:02.572433007 +0000 UTC m=+1654.812668120" Oct 30 01:00:03.166174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.166135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:03.441429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.441401 5938 generic.go:356] "Generic (PLEG): container finished" podID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerID="85d5d55507cc4eafb7a0362a198cda8dac1b556d636caf6d829191c43deaa62f" exitCode=60 Oct 30 01:00:03.441564 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.441540 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerDied","Data":"85d5d55507cc4eafb7a0362a198cda8dac1b556d636caf6d829191c43deaa62f"} Oct 30 01:00:03.442068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.442047 5938 scope.go:122] "RemoveContainer" containerID="85d5d55507cc4eafb7a0362a198cda8dac1b556d636caf6d829191c43deaa62f" Oct 30 01:00:03.443659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.443605 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerStarted","Data":"b339df9e165c951e0c5c81555ccfaeaabd2f88754876d8799831b1730581f4de"} Oct 30 01:00:03.446161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.446135 5938 generic.go:356] "Generic (PLEG): container finished" podID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerID="fb09055c87891191801e3df25c32dbf3173a750707b3404b5e34fa61a16b9e43" exitCode=60 Oct 30 01:00:03.446284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.446214 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerDied","Data":"fb09055c87891191801e3df25c32dbf3173a750707b3404b5e34fa61a16b9e43"} Oct 30 01:00:03.446924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.446904 5938 scope.go:122] "RemoveContainer" containerID="fb09055c87891191801e3df25c32dbf3173a750707b3404b5e34fa61a16b9e43" Oct 30 01:00:03.448299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.448274 5938 generic.go:356] "Generic (PLEG): container finished" podID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerID="d93f5b8e5800811a9ed16b91ef9f5ea26aedb7471ebab4f37fddc37d62aae29c" exitCode=60 Oct 30 01:00:03.448386 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.448354 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerDied","Data":"d93f5b8e5800811a9ed16b91ef9f5ea26aedb7471ebab4f37fddc37d62aae29c"} Oct 30 01:00:03.448615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.448599 5938 scope.go:122] "RemoveContainer" containerID="d93f5b8e5800811a9ed16b91ef9f5ea26aedb7471ebab4f37fddc37d62aae29c" Oct 30 01:00:03.450224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.450201 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerID="de2f32e8650d5dc9a86fe0adabf636c37d5e94bc4657cc3f326f52b7b1ea04c1" exitCode=60 Oct 30 01:00:03.450336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.450229 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerDied","Data":"de2f32e8650d5dc9a86fe0adabf636c37d5e94bc4657cc3f326f52b7b1ea04c1"} Oct 30 01:00:03.450534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.450517 5938 scope.go:122] "RemoveContainer" containerID="de2f32e8650d5dc9a86fe0adabf636c37d5e94bc4657cc3f326f52b7b1ea04c1" Oct 30 01:00:03.452178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.452157 5938 generic.go:356] "Generic (PLEG): container finished" podID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerID="c224a2e34027fc822fd19c5dcd41aa576c8de5306cad7ed01ab4f1ecfe954564" exitCode=60 Oct 30 01:00:03.452286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.452238 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerDied","Data":"c224a2e34027fc822fd19c5dcd41aa576c8de5306cad7ed01ab4f1ecfe954564"} Oct 30 01:00:03.452435 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.452420 5938 scope.go:122] "RemoveContainer" containerID="c224a2e34027fc822fd19c5dcd41aa576c8de5306cad7ed01ab4f1ecfe954564" Oct 30 01:00:03.463024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.462878 5938 generic.go:356] "Generic (PLEG): container finished" podID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerID="7c08e7a59f7106f246b3c04657cdd29577ba63b12cc98ef662fc1274fc4b9a6a" exitCode=60 Oct 30 01:00:03.463024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.462988 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerDied","Data":"7c08e7a59f7106f246b3c04657cdd29577ba63b12cc98ef662fc1274fc4b9a6a"} Oct 30 01:00:03.463393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.463232 5938 scope.go:122] "RemoveContainer" containerID="7c08e7a59f7106f246b3c04657cdd29577ba63b12cc98ef662fc1274fc4b9a6a" Oct 30 01:00:03.466043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.466022 5938 generic.go:356] "Generic (PLEG): container finished" podID="9c87f67e-8633-4511-954e-9d38b1119b97" containerID="a160cac8a53d67f9322d2672541fa9597894db2ccff8e5fdd87d4ee37a465c33" exitCode=60 Oct 30 01:00:03.466135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.466118 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerDied","Data":"a160cac8a53d67f9322d2672541fa9597894db2ccff8e5fdd87d4ee37a465c33"} Oct 30 01:00:03.466379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.466359 5938 scope.go:122] "RemoveContainer" containerID="a160cac8a53d67f9322d2672541fa9597894db2ccff8e5fdd87d4ee37a465c33" Oct 30 01:00:03.468473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.468453 5938 generic.go:356] "Generic (PLEG): container finished" podID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerID="fda15610e9a4704784d9916e2403379aea06eb48d2e4515607e4b1aa8f9ebdf4" exitCode=60 Oct 30 01:00:03.468584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.468540 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerDied","Data":"fda15610e9a4704784d9916e2403379aea06eb48d2e4515607e4b1aa8f9ebdf4"} Oct 30 01:00:03.468775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.468761 5938 scope.go:122] "RemoveContainer" containerID="fda15610e9a4704784d9916e2403379aea06eb48d2e4515607e4b1aa8f9ebdf4" Oct 30 01:00:03.470454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.470429 5938 generic.go:356] "Generic (PLEG): container finished" podID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerID="b02d6ed4e01f8dc15e1bf6b58b7a5af82702faf58ce457299e760bf2298c30ea" exitCode=60 Oct 30 01:00:03.470529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.470466 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerDied","Data":"b02d6ed4e01f8dc15e1bf6b58b7a5af82702faf58ce457299e760bf2298c30ea"} Oct 30 01:00:03.470707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.470691 5938 scope.go:122] "RemoveContainer" containerID="b02d6ed4e01f8dc15e1bf6b58b7a5af82702faf58ce457299e760bf2298c30ea" Oct 30 01:00:03.728104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.728031 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" podStartSLOduration=3.728010014 podStartE2EDuration="3.728010014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.720720414 +0000 UTC m=+1655.960955527" watchObservedRunningTime="2026-10-30 01:00:03.728010014 +0000 UTC m=+1655.968245129" Oct 30 01:00:03.786256 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:03.786223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:04.482031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.481990 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerStarted","Data":"82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350"} Oct 30 01:00:04.488122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.488085 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerStarted","Data":"7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0"} Oct 30 01:00:04.491173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.491143 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerStarted","Data":"2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80"} Oct 30 01:00:04.493506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.493482 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerStarted","Data":"7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753"} Oct 30 01:00:04.495927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.495898 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerStarted","Data":"8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8"} Oct 30 01:00:04.498095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.498070 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerStarted","Data":"bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142"} Oct 30 01:00:04.499676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.499654 5938 generic.go:356] "Generic (PLEG): container finished" podID="a2ea8907-88e5-4304-8044-717f0f87493b" containerID="b339df9e165c951e0c5c81555ccfaeaabd2f88754876d8799831b1730581f4de" exitCode=60 Oct 30 01:00:04.499784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.499736 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerDied","Data":"b339df9e165c951e0c5c81555ccfaeaabd2f88754876d8799831b1730581f4de"} Oct 30 01:00:04.499930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.499911 5938 scope.go:122] "RemoveContainer" containerID="b339df9e165c951e0c5c81555ccfaeaabd2f88754876d8799831b1730581f4de" Oct 30 01:00:04.502003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.501982 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerStarted","Data":"7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55"} Oct 30 01:00:04.504014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.503987 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerStarted","Data":"07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0"} Oct 30 01:00:04.505997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.505974 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerStarted","Data":"f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416"} Oct 30 01:00:04.521946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.521898 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" podStartSLOduration=4.52187979 podStartE2EDuration="4.52187979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.520157392 +0000 UTC m=+1656.760392506" watchObservedRunningTime="2026-10-30 01:00:04.52187979 +0000 UTC m=+1656.762114910" Oct 30 01:00:04.680666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.680615 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" podStartSLOduration=4.680600515 podStartE2EDuration="4.680600515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.67949003 +0000 UTC m=+1656.919725158" watchObservedRunningTime="2026-10-30 01:00:04.680600515 +0000 UTC m=+1656.920835629" Oct 30 01:00:04.725799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.725751 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" podStartSLOduration=4.725737561 podStartE2EDuration="4.725737561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.721717745 +0000 UTC m=+1656.961952859" watchObservedRunningTime="2026-10-30 01:00:04.725737561 +0000 UTC m=+1656.965972674" Oct 30 01:00:04.772109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.772069 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:04.802723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.802634 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" podStartSLOduration=4.802617223 podStartE2EDuration="4.802617223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.800205189 +0000 UTC m=+1657.040440302" watchObservedRunningTime="2026-10-30 01:00:04.802617223 +0000 UTC m=+1657.042852337" Oct 30 01:00:04.984503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:04.984076 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" podStartSLOduration=4.984057108 podStartE2EDuration="4.984057108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:04.982187047 +0000 UTC m=+1657.222422161" watchObservedRunningTime="2026-10-30 01:00:04.984057108 +0000 UTC m=+1657.224292222" Oct 30 01:00:05.020162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.020132 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.180856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.180827 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.522146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.522107 5938 generic.go:356] "Generic (PLEG): container finished" podID="9c87f67e-8633-4511-954e-9d38b1119b97" containerID="2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80" exitCode=60 Oct 30 01:00:05.522614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.522241 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerDied","Data":"2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80"} Oct 30 01:00:05.522614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.522298 5938 scope.go:122] "RemoveContainer" containerID="a160cac8a53d67f9322d2672541fa9597894db2ccff8e5fdd87d4ee37a465c33" Oct 30 01:00:05.522837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.522818 5938 scope.go:122] "RemoveContainer" containerID="2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80" Oct 30 01:00:05.523301 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.523268 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb_odh-ai-gateway-infra(9c87f67e-8633-4511-954e-9d38b1119b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" Oct 30 01:00:05.526937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.526907 5938 generic.go:356] "Generic (PLEG): container finished" podID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerID="7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753" exitCode=60 Oct 30 01:00:05.527085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.526967 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerDied","Data":"7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753"} Oct 30 01:00:05.527275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.527259 5938 scope.go:122] "RemoveContainer" containerID="7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753" Oct 30 01:00:05.527623 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.527599 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs_odh-ai-gateway-infra(72523efe-c57d-43c5-b44a-a3aa7edea5d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" Oct 30 01:00:05.530198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.529389 5938 generic.go:356] "Generic (PLEG): container finished" podID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerID="8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8" exitCode=60 Oct 30 01:00:05.530198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.529529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerDied","Data":"8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8"} Oct 30 01:00:05.530198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.529758 5938 scope.go:122] "RemoveContainer" containerID="8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8" Oct 30 01:00:05.530198 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.530127 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v_odh-ai-gateway-infra(da88969c-2f54-4e2e-be1f-9c2491b432c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" Oct 30 01:00:05.532197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.532177 5938 generic.go:356] "Generic (PLEG): container finished" podID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerID="bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142" exitCode=60 Oct 30 01:00:05.532312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.532262 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerDied","Data":"bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142"} Oct 30 01:00:05.532588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.532568 5938 scope.go:122] "RemoveContainer" containerID="bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142" Oct 30 01:00:05.532986 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.532960 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm_odh-ai-gateway-infra(79014303-b1c3-4d3a-970f-bee9997cc69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" Oct 30 01:00:05.534700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.534675 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerStarted","Data":"56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe"} Oct 30 01:00:05.536531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.536510 5938 generic.go:356] "Generic (PLEG): container finished" podID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerID="7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55" exitCode=60 Oct 30 01:00:05.536638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.536575 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerDied","Data":"7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55"} Oct 30 01:00:05.536963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.536939 5938 scope.go:122] "RemoveContainer" containerID="7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55" Oct 30 01:00:05.537264 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.537242 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx_odh-ai-gateway-infra(16f41ef4-7167-4e1f-8c92-6b37b49df4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" Oct 30 01:00:05.540397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.540375 5938 generic.go:356] "Generic (PLEG): container finished" podID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerID="07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0" exitCode=60 Oct 30 01:00:05.540493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.540448 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerDied","Data":"07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0"} Oct 30 01:00:05.540815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.540798 5938 scope.go:122] "RemoveContainer" containerID="07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0" Oct 30 01:00:05.541179 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.541143 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w_odh-ai-gateway-infra(37221d5b-9f20-4a94-bbc3-6e5837f058c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" Oct 30 01:00:05.542935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.542915 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerID="f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416" exitCode=60 Oct 30 01:00:05.543167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.543058 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerDied","Data":"f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416"} Oct 30 01:00:05.543238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.543180 5938 scope.go:122] "RemoveContainer" containerID="f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416" Oct 30 01:00:05.543287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.543234 5938 scope.go:122] "RemoveContainer" containerID="fda15610e9a4704784d9916e2403379aea06eb48d2e4515607e4b1aa8f9ebdf4" Oct 30 01:00:05.543467 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.543443 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5_odh-ai-gateway-infra(8a41fb40-5144-4e93-92b4-3dec7d41aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" Oct 30 01:00:05.545101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.545081 5938 generic.go:356] "Generic (PLEG): container finished" podID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerID="82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350" exitCode=60 Oct 30 01:00:05.545194 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.545180 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerDied","Data":"82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350"} Oct 30 01:00:05.545562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.545326 5938 scope.go:122] "RemoveContainer" containerID="82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350" Oct 30 01:00:05.545806 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.545781 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p_odh-ai-gateway-infra(641aa109-dc7d-4d4e-ba48-a186b6d49a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" Oct 30 01:00:05.547378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.547356 5938 generic.go:356] "Generic (PLEG): container finished" podID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerID="7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0" exitCode=60 Oct 30 01:00:05.547453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.547418 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerDied","Data":"7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0"} Oct 30 01:00:05.547668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.547655 5938 scope.go:122] "RemoveContainer" containerID="7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0" Oct 30 01:00:05.547930 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:05.547910 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585_odh-ai-gateway-infra(d7b942c3-53a0-4482-a105-25303ffea6d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" Oct 30 01:00:05.559436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.559417 5938 scope.go:122] "RemoveContainer" containerID="b02d6ed4e01f8dc15e1bf6b58b7a5af82702faf58ce457299e760bf2298c30ea" Oct 30 01:00:05.580934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.580907 5938 scope.go:122] "RemoveContainer" containerID="85d5d55507cc4eafb7a0362a198cda8dac1b556d636caf6d829191c43deaa62f" Oct 30 01:00:05.679717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.679690 5938 scope.go:122] "RemoveContainer" containerID="fb09055c87891191801e3df25c32dbf3173a750707b3404b5e34fa61a16b9e43" Oct 30 01:00:05.698160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.697247 5938 scope.go:122] "RemoveContainer" containerID="d93f5b8e5800811a9ed16b91ef9f5ea26aedb7471ebab4f37fddc37d62aae29c" Oct 30 01:00:05.711431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.711411 5938 scope.go:122] "RemoveContainer" containerID="de2f32e8650d5dc9a86fe0adabf636c37d5e94bc4657cc3f326f52b7b1ea04c1" Oct 30 01:00:05.727883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.727853 5938 scope.go:122] "RemoveContainer" containerID="c224a2e34027fc822fd19c5dcd41aa576c8de5306cad7ed01ab4f1ecfe954564" Oct 30 01:00:05.743143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:05.743126 5938 scope.go:122] "RemoveContainer" containerID="7c08e7a59f7106f246b3c04657cdd29577ba63b12cc98ef662fc1274fc4b9a6a" Oct 30 01:00:06.401090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.401055 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:06.603282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.603249 5938 scope.go:122] "RemoveContainer" containerID="bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142" Oct 30 01:00:06.604328 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.604291 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm_odh-ai-gateway-infra(79014303-b1c3-4d3a-970f-bee9997cc69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" Oct 30 01:00:06.612740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.612711 5938 generic.go:356] "Generic (PLEG): container finished" podID="a2ea8907-88e5-4304-8044-717f0f87493b" containerID="56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe" exitCode=60 Oct 30 01:00:06.613008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.612987 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerDied","Data":"56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe"} Oct 30 01:00:06.613414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.613342 5938 scope.go:122] "RemoveContainer" containerID="b339df9e165c951e0c5c81555ccfaeaabd2f88754876d8799831b1730581f4de" Oct 30 01:00:06.614239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.614209 5938 scope.go:122] "RemoveContainer" containerID="56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe" Oct 30 01:00:06.614793 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.614756 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6_odh-ai-gateway-infra(a2ea8907-88e5-4304-8044-717f0f87493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" Oct 30 01:00:06.644854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.641803 5938 scope.go:122] "RemoveContainer" containerID="7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55" Oct 30 01:00:06.644854 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.642142 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx_odh-ai-gateway-infra(16f41ef4-7167-4e1f-8c92-6b37b49df4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" Oct 30 01:00:06.646133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.645666 5938 scope.go:122] "RemoveContainer" containerID="07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0" Oct 30 01:00:06.646258 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.646069 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w_odh-ai-gateway-infra(37221d5b-9f20-4a94-bbc3-6e5837f058c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" Oct 30 01:00:06.685410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.682332 5938 scope.go:122] "RemoveContainer" containerID="f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416" Oct 30 01:00:06.685410 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.682814 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5_odh-ai-gateway-infra(8a41fb40-5144-4e93-92b4-3dec7d41aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" Oct 30 01:00:06.685410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.684733 5938 scope.go:122] "RemoveContainer" containerID="82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350" Oct 30 01:00:06.685410 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.685139 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p_odh-ai-gateway-infra(641aa109-dc7d-4d4e-ba48-a186b6d49a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" Oct 30 01:00:06.689141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.689120 5938 scope.go:122] "RemoveContainer" containerID="7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0" Oct 30 01:00:06.689655 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.689630 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585_odh-ai-gateway-infra(d7b942c3-53a0-4482-a105-25303ffea6d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" Oct 30 01:00:06.691270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.691254 5938 scope.go:122] "RemoveContainer" containerID="2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80" Oct 30 01:00:06.691682 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.691659 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb_odh-ai-gateway-infra(9c87f67e-8633-4511-954e-9d38b1119b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" Oct 30 01:00:06.693902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.693885 5938 scope.go:122] "RemoveContainer" containerID="7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753" Oct 30 01:00:06.694721 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.694697 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs_odh-ai-gateway-infra(72523efe-c57d-43c5-b44a-a3aa7edea5d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" Oct 30 01:00:06.705421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:06.705396 5938 scope.go:122] "RemoveContainer" containerID="8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8" Oct 30 01:00:06.705748 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:06.705721 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v_odh-ai-gateway-infra(da88969c-2f54-4e2e-be1f-9c2491b432c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" Oct 30 01:00:07.095190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.095095 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:07.715465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.715404 5938 generic.go:356] "Generic (PLEG): container finished" podID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerID="a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6" exitCode=0 Oct 30 01:00:07.715901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.715477 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerDied","Data":"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6"} Oct 30 01:00:07.715967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.715931 5938 scope.go:122] "RemoveContainer" containerID="a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6" Oct 30 01:00:07.717802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.717782 5938 scope.go:122] "RemoveContainer" containerID="56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe" Oct 30 01:00:07.717941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.717914 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:07.718012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.717995 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:07.718055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:07.718044 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:07.718152 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:07.718131 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6_odh-ai-gateway-infra(a2ea8907-88e5-4304-8044-717f0f87493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" Oct 30 01:00:08.279814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:08.279776 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:08.730416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:08.730370 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerStarted","Data":"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5"} Oct 30 01:00:08.922800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:08.922760 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:09.348541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:09.348507 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:00:09.350739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:09.350711 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 01:00:09.350903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:09.350766 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 01:00:10.353463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:10.350100 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 01:00:10.353463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:10.350162 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 01:00:10.799514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:10.784188 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:11.349511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:11.349479 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 01:00:11.349699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:11.349532 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 01:00:11.762079 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:11.762045 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:11.910200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:11.910156 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:12.349270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:12.349231 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 01:00:12.349486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:12.349292 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 01:00:12.478889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:12.478834 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:13.349485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:13.349450 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 01:00:13.349916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:13.349508 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 01:00:14.308001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:14.307968 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:14.444114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:14.444083 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:00:14.445397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:14.445369 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:00:14.446121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:14.446102 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:00:15.398936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:15.398897 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:15.869137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:15.869108 5938 scope.go:122] "RemoveContainer" containerID="07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0" Oct 30 01:00:15.869603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:15.869250 5938 scope.go:122] "RemoveContainer" containerID="f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416" Oct 30 01:00:15.869603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:15.869327 5938 scope.go:122] "RemoveContainer" containerID="2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80" Oct 30 01:00:16.224019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:16.223984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:16.812830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:16.812799 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerStarted","Data":"c046ab3f5dcef3eebd0a245844db3d2d4dac76743684e86ed7df597dea31dd06"} Oct 30 01:00:16.814764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:16.814738 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerStarted","Data":"8288c6a690f84e2a7b8762c9f6e7513597b4c0c482b93b9f895fd80b3f979d45"} Oct 30 01:00:16.816730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:16.816704 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerStarted","Data":"a2758f67f2ec35040c89964a0950e804e545bd38a08ff68c1acc27a56b75f026"} Oct 30 01:00:16.869232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:16.869209 5938 scope.go:122] "RemoveContainer" containerID="7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0" Oct 30 01:00:17.834552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.834516 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerStarted","Data":"1b7418fc957c02d424289c05bf665627187ca49b3b73c2f05a6cbda0fcecd7da"} Oct 30 01:00:17.836826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.836801 5938 generic.go:356] "Generic (PLEG): container finished" podID="9c87f67e-8633-4511-954e-9d38b1119b97" containerID="a2758f67f2ec35040c89964a0950e804e545bd38a08ff68c1acc27a56b75f026" exitCode=60 Oct 30 01:00:17.836986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.836885 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerDied","Data":"a2758f67f2ec35040c89964a0950e804e545bd38a08ff68c1acc27a56b75f026"} Oct 30 01:00:17.836986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.836941 5938 scope.go:122] "RemoveContainer" containerID="2d9a9f2270ac3fa3ebefe6adb48c01c4fa6038c8f43e12a939a0acc8469e4e80" Oct 30 01:00:17.837320 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.837303 5938 scope.go:122] "RemoveContainer" containerID="a2758f67f2ec35040c89964a0950e804e545bd38a08ff68c1acc27a56b75f026" Oct 30 01:00:17.837726 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:17.837698 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb_odh-ai-gateway-infra(9c87f67e-8633-4511-954e-9d38b1119b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" Oct 30 01:00:17.840382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.840360 5938 generic.go:356] "Generic (PLEG): container finished" podID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerID="c046ab3f5dcef3eebd0a245844db3d2d4dac76743684e86ed7df597dea31dd06" exitCode=60 Oct 30 01:00:17.840473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.840433 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerDied","Data":"c046ab3f5dcef3eebd0a245844db3d2d4dac76743684e86ed7df597dea31dd06"} Oct 30 01:00:17.840888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.840859 5938 scope.go:122] "RemoveContainer" containerID="c046ab3f5dcef3eebd0a245844db3d2d4dac76743684e86ed7df597dea31dd06" Oct 30 01:00:17.841209 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:17.841191 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w_odh-ai-gateway-infra(37221d5b-9f20-4a94-bbc3-6e5837f058c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" Oct 30 01:00:17.842544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.842525 5938 generic.go:356] "Generic (PLEG): container finished" podID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerID="8288c6a690f84e2a7b8762c9f6e7513597b4c0c482b93b9f895fd80b3f979d45" exitCode=60 Oct 30 01:00:17.842637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.842596 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerDied","Data":"8288c6a690f84e2a7b8762c9f6e7513597b4c0c482b93b9f895fd80b3f979d45"} Oct 30 01:00:17.843021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.843004 5938 scope.go:122] "RemoveContainer" containerID="8288c6a690f84e2a7b8762c9f6e7513597b4c0c482b93b9f895fd80b3f979d45" Oct 30 01:00:17.843302 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:17.843283 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5_odh-ai-gateway-infra(8a41fb40-5144-4e93-92b4-3dec7d41aab1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" Oct 30 01:00:17.850998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.850971 5938 scope.go:122] "RemoveContainer" containerID="07d554fe165ea0c00e7cff7d5bc93ab22832be00ccc33fc917367e380d7b97d0" Oct 30 01:00:17.863736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.863712 5938 scope.go:122] "RemoveContainer" containerID="f92bf4b96007879cd6023e26ad2e1f51e135f2016c21954a5ba38bd8e384d416" Oct 30 01:00:17.868176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.868157 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:17.883501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.883480 5938 scope.go:122] "RemoveContainer" containerID="7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55" Oct 30 01:00:17.883874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.883646 5938 scope.go:122] "RemoveContainer" containerID="82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350" Oct 30 01:00:17.883874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.883851 5938 scope.go:122] "RemoveContainer" containerID="7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753" Oct 30 01:00:17.905449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.905427 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:17.944487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:17.944462 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:18.047658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.044769 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:18.854780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.854747 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerStarted","Data":"4be0cf4b2864216ea25fbccc78a5a0c6c65a36d8d196e6f31e98858cd9469e11"} Oct 30 01:00:18.856948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.856921 5938 generic.go:356] "Generic (PLEG): container finished" podID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerID="1b7418fc957c02d424289c05bf665627187ca49b3b73c2f05a6cbda0fcecd7da" exitCode=60 Oct 30 01:00:18.857068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.856998 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerDied","Data":"1b7418fc957c02d424289c05bf665627187ca49b3b73c2f05a6cbda0fcecd7da"} Oct 30 01:00:18.857068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.857057 5938 scope.go:122] "RemoveContainer" containerID="7d2f27e2d7ad68ff1b337fd9c1c23d2b96de79974dcdf9ccfc688159bdccdad0" Oct 30 01:00:18.857372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.857351 5938 scope.go:122] "RemoveContainer" containerID="1b7418fc957c02d424289c05bf665627187ca49b3b73c2f05a6cbda0fcecd7da" Oct 30 01:00:18.857710 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:18.857673 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585_odh-ai-gateway-infra(d7b942c3-53a0-4482-a105-25303ffea6d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" Oct 30 01:00:18.862497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.862471 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerStarted","Data":"d085765a463a8780df0a2623b383b3fbd4137149d45f0598e15c8e7ee2057fa3"} Oct 30 01:00:18.865111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.865088 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerStarted","Data":"caf9528c707bef79da682dbaa2188173fe298ea2050bcdbbf604f46237340f8d"} Oct 30 01:00:18.871911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.868663 5938 scope.go:122] "RemoveContainer" containerID="8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8" Oct 30 01:00:18.871911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.869362 5938 scope.go:122] "RemoveContainer" containerID="bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142" Oct 30 01:00:18.871911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.869686 5938 scope.go:122] "RemoveContainer" containerID="56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe" Oct 30 01:00:18.902263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:18.901466 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:19.375225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.375200 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:19.380118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.380100 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:19.384019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.383998 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:19.440158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440081 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\") pod \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " Oct 30 01:00:19.440158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440135 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\") pod \"9c87f67e-8633-4511-954e-9d38b1119b97\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " Oct 30 01:00:19.440368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\" (UniqueName: \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\") pod \"9c87f67e-8633-4511-954e-9d38b1119b97\" (UID: \"9c87f67e-8633-4511-954e-9d38b1119b97\") " Oct 30 01:00:19.440368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440222 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\" (UniqueName: \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\") pod \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " Oct 30 01:00:19.440496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440364 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\") pod \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\" (UID: \"37221d5b-9f20-4a94-bbc3-6e5837f058c9\") " Oct 30 01:00:19.440496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440419 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\" (UniqueName: \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\") pod \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\" (UID: \"8a41fb40-5144-4e93-92b4-3dec7d41aab1\") " Oct 30 01:00:19.440604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440502 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca" pod "9c87f67e-8633-4511-954e-9d38b1119b97" (UID: "9c87f67e-8633-4511-954e-9d38b1119b97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.441400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.440506 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca" pod "8a41fb40-5144-4e93-92b4-3dec7d41aab1" (UID: "8a41fb40-5144-4e93-92b4-3dec7d41aab1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.441400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.441014 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca" pod "37221d5b-9f20-4a94-bbc3-6e5837f058c9" (UID: "37221d5b-9f20-4a94-bbc3-6e5837f058c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:19.441400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.441360 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37221d5b-9f20-4a94-bbc3-6e5837f058c9-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.441400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.441374 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a41fb40-5144-4e93-92b4-3dec7d41aab1-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.441400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.441383 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c87f67e-8633-4511-954e-9d38b1119b97-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.488172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.487594 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf" pod "37221d5b-9f20-4a94-bbc3-6e5837f058c9" (UID: "37221d5b-9f20-4a94-bbc3-6e5837f058c9"). InnerVolumeSpecName "kube-api-access-4bnjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.488172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.488110 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt" pod "8a41fb40-5144-4e93-92b4-3dec7d41aab1" (UID: "8a41fb40-5144-4e93-92b4-3dec7d41aab1"). InnerVolumeSpecName "kube-api-access-9z8qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.489007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.488142 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59" pod "9c87f67e-8633-4511-954e-9d38b1119b97" (UID: "9c87f67e-8633-4511-954e-9d38b1119b97"). InnerVolumeSpecName "kube-api-access-hzx59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:19.542644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.542613 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8qt\" (UniqueName: \"kubernetes.io/projected/8a41fb40-5144-4e93-92b4-3dec7d41aab1-kube-api-access-9z8qt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.542644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.542643 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzx59\" (UniqueName: \"kubernetes.io/projected/9c87f67e-8633-4511-954e-9d38b1119b97-kube-api-access-hzx59\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.542847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.542657 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnjf\" (UniqueName: \"kubernetes.io/projected/37221d5b-9f20-4a94-bbc3-6e5837f058c9-kube-api-access-4bnjf\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:19.883633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.883608 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" Oct 30 01:00:19.886500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.886461 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb" event={"ID":"9c87f67e-8633-4511-954e-9d38b1119b97","Type":"ContainerDied","Data":"d8dc3fab6463a5fed395974faee9dd952e6741334db23c40f28b230877a1bea6"} Oct 30 01:00:19.886645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.886522 5938 scope.go:122] "RemoveContainer" containerID="a2758f67f2ec35040c89964a0950e804e545bd38a08ff68c1acc27a56b75f026" Oct 30 01:00:19.896080 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.896049 5938 generic.go:356] "Generic (PLEG): container finished" podID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerID="d085765a463a8780df0a2623b383b3fbd4137149d45f0598e15c8e7ee2057fa3" exitCode=60 Oct 30 01:00:19.896164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.896150 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerDied","Data":"d085765a463a8780df0a2623b383b3fbd4137149d45f0598e15c8e7ee2057fa3"} Oct 30 01:00:19.896610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.896587 5938 scope.go:122] "RemoveContainer" containerID="d085765a463a8780df0a2623b383b3fbd4137149d45f0598e15c8e7ee2057fa3" Oct 30 01:00:19.896994 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:19.896961 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs_odh-ai-gateway-infra(72523efe-c57d-43c5-b44a-a3aa7edea5d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" Oct 30 01:00:19.899973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.899950 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerStarted","Data":"174f8ba987c17774e568a852c283d3c4e980da03465d3e8c9c8c05d27c9361b9"} Oct 30 01:00:19.902467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.902440 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerStarted","Data":"57fa3a30bb01a26c20cd4798bec7bb0514104bc146e55c347b9ce97d48e8d062"} Oct 30 01:00:19.904659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.904634 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerStarted","Data":"2493c10eefb6eb00620c731144029b78fb5074c5be69a2547cbf385e224fa75c"} Oct 30 01:00:19.906713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.906685 5938 generic.go:356] "Generic (PLEG): container finished" podID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerID="caf9528c707bef79da682dbaa2188173fe298ea2050bcdbbf604f46237340f8d" exitCode=60 Oct 30 01:00:19.906842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.906761 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerDied","Data":"caf9528c707bef79da682dbaa2188173fe298ea2050bcdbbf604f46237340f8d"} Oct 30 01:00:19.907141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.907124 5938 scope.go:122] "RemoveContainer" containerID="caf9528c707bef79da682dbaa2188173fe298ea2050bcdbbf604f46237340f8d" Oct 30 01:00:19.907357 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:19.907339 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx_odh-ai-gateway-infra(16f41ef4-7167-4e1f-8c92-6b37b49df4fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" Oct 30 01:00:19.908673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.908652 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" event={"ID":"37221d5b-9f20-4a94-bbc3-6e5837f058c9","Type":"ContainerDied","Data":"099467fcf9bb5200659c553a7bd2e28631abd124f0343ebab1927166c2f1d255"} Oct 30 01:00:19.908780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.908660 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w" Oct 30 01:00:19.910174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.910157 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" Oct 30 01:00:19.910268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.910200 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5" event={"ID":"8a41fb40-5144-4e93-92b4-3dec7d41aab1","Type":"ContainerDied","Data":"eb1cf012a4b341c013287af66f974e62f21da4a89c1cad5816c3ddf4f0ff40fa"} Oct 30 01:00:19.913643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.913627 5938 scope.go:122] "RemoveContainer" containerID="7715397714c9a46c810ba8b5dc9e39e1027ea3f64180f926f2dc1d179191a753" Oct 30 01:00:19.914156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.914136 5938 generic.go:356] "Generic (PLEG): container finished" podID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerID="4be0cf4b2864216ea25fbccc78a5a0c6c65a36d8d196e6f31e98858cd9469e11" exitCode=60 Oct 30 01:00:19.914270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.914166 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerDied","Data":"4be0cf4b2864216ea25fbccc78a5a0c6c65a36d8d196e6f31e98858cd9469e11"} Oct 30 01:00:19.914456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.914442 5938 scope.go:122] "RemoveContainer" containerID="4be0cf4b2864216ea25fbccc78a5a0c6c65a36d8d196e6f31e98858cd9469e11" Oct 30 01:00:19.914702 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:19.914683 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p_odh-ai-gateway-infra(641aa109-dc7d-4d4e-ba48-a186b6d49a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" Oct 30 01:00:19.930391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.930376 5938 scope.go:122] "RemoveContainer" containerID="7ac63f4c835efa5a092a99a17573e6ae0421a1c10a209360d20affe301d06b55" Oct 30 01:00:19.952037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.952012 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:19.957013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.956849 5938 scope.go:122] "RemoveContainer" containerID="c046ab3f5dcef3eebd0a245844db3d2d4dac76743684e86ed7df597dea31dd06" Oct 30 01:00:19.977227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.977207 5938 scope.go:122] "RemoveContainer" containerID="8288c6a690f84e2a7b8762c9f6e7513597b4c0c482b93b9f895fd80b3f979d45" Oct 30 01:00:19.997485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:19.997215 5938 scope.go:122] "RemoveContainer" containerID="82379f4aeeaea372c72cc2bce9b187aedd33848bfb672df09c3be9aa96517350" Oct 30 01:00:20.080969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.077559 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:20.080969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.078154 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:20.130195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.130162 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:20.135592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.135365 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6444d7-29888700-j6zq5"] Oct 30 01:00:20.141468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.141443 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:20.152243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.152210 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\" (UniqueName: \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\") pod \"d7b942c3-53a0-4482-a105-25303ffea6d7\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " Oct 30 01:00:20.152412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.152298 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\") pod \"d7b942c3-53a0-4482-a105-25303ffea6d7\" (UID: \"d7b942c3-53a0-4482-a105-25303ffea6d7\") " Oct 30 01:00:20.153642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.153610 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca" pod "d7b942c3-53a0-4482-a105-25303ffea6d7" (UID: "d7b942c3-53a0-4482-a105-25303ffea6d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:20.157541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.157517 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv" pod "d7b942c3-53a0-4482-a105-25303ffea6d7" (UID: "d7b942c3-53a0-4482-a105-25303ffea6d7"). InnerVolumeSpecName "kube-api-access-v2cvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:20.163486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.163460 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:20.170858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.170827 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-51b95e-29888700-xtpdb"] Oct 30 01:00:20.180310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.180164 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:20.243833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.243806 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:20.247726 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.247704 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-03d892-29888700-xff9w"] Oct 30 01:00:20.255047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.255023 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cvv\" (UniqueName: \"kubernetes.io/projected/d7b942c3-53a0-4482-a105-25303ffea6d7-kube-api-access-v2cvv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.255138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.255049 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b942c3-53a0-4482-a105-25303ffea6d7-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:20.896819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.896792 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:20.927685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.927661 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" Oct 30 01:00:20.928175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.927682 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585" event={"ID":"d7b942c3-53a0-4482-a105-25303ffea6d7","Type":"ContainerDied","Data":"24c28a5339b67c02bc842e00a9a937ed5ba478562bd732f6298ffac7701c9073"} Oct 30 01:00:20.928175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.927741 5938 scope.go:122] "RemoveContainer" containerID="1b7418fc957c02d424289c05bf665627187ca49b3b73c2f05a6cbda0fcecd7da" Oct 30 01:00:20.937062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.937032 5938 generic.go:356] "Generic (PLEG): container finished" podID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerID="174f8ba987c17774e568a852c283d3c4e980da03465d3e8c9c8c05d27c9361b9" exitCode=60 Oct 30 01:00:20.937169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.937106 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerDied","Data":"174f8ba987c17774e568a852c283d3c4e980da03465d3e8c9c8c05d27c9361b9"} Oct 30 01:00:20.937578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.937558 5938 scope.go:122] "RemoveContainer" containerID="174f8ba987c17774e568a852c283d3c4e980da03465d3e8c9c8c05d27c9361b9" Oct 30 01:00:20.937998 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:20.937970 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v_odh-ai-gateway-infra(da88969c-2f54-4e2e-be1f-9c2491b432c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" Oct 30 01:00:20.939580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.939558 5938 generic.go:356] "Generic (PLEG): container finished" podID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerID="57fa3a30bb01a26c20cd4798bec7bb0514104bc146e55c347b9ce97d48e8d062" exitCode=60 Oct 30 01:00:20.939689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.939624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerDied","Data":"57fa3a30bb01a26c20cd4798bec7bb0514104bc146e55c347b9ce97d48e8d062"} Oct 30 01:00:20.940085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.940065 5938 scope.go:122] "RemoveContainer" containerID="57fa3a30bb01a26c20cd4798bec7bb0514104bc146e55c347b9ce97d48e8d062" Oct 30 01:00:20.940427 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:20.940408 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm_odh-ai-gateway-infra(79014303-b1c3-4d3a-970f-bee9997cc69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" Oct 30 01:00:20.941938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.941913 5938 generic.go:356] "Generic (PLEG): container finished" podID="a2ea8907-88e5-4304-8044-717f0f87493b" containerID="2493c10eefb6eb00620c731144029b78fb5074c5be69a2547cbf385e224fa75c" exitCode=60 Oct 30 01:00:20.942040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.941995 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerDied","Data":"2493c10eefb6eb00620c731144029b78fb5074c5be69a2547cbf385e224fa75c"} Oct 30 01:00:20.942514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.942369 5938 scope.go:122] "RemoveContainer" containerID="2493c10eefb6eb00620c731144029b78fb5074c5be69a2547cbf385e224fa75c" Oct 30 01:00:20.942742 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:20.942719 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6_odh-ai-gateway-infra(a2ea8907-88e5-4304-8044-717f0f87493b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" Oct 30 01:00:20.958372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.958345 5938 scope.go:122] "RemoveContainer" containerID="8e09ea79e7434662f149588f1f1d863b32d55fc9eeed273decb972b8a7e8fef8" Oct 30 01:00:20.971672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:20.971186 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:21.010182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.010157 5938 scope.go:122] "RemoveContainer" containerID="bd595e25c767a92f7b2f3fdc68a115256a4605fc4b78fe63dcaa0166f0b1a142" Oct 30 01:00:21.037635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.037611 5938 scope.go:122] "RemoveContainer" containerID="56a1e935d0a8e12dd6aba6c21568523086e6d62cad99e6cea245fd9498e728fe" Oct 30 01:00:21.059707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.059606 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:21.197799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.197770 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:21.213720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.213688 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:21.218600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.218575 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-84b89f-29888700-zw585"] Oct 30 01:00:21.225127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.225100 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:21.269535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.269514 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\" (UniqueName: \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\") pod \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " Oct 30 01:00:21.269725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.269704 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\") pod \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\" (UID: \"641aa109-dc7d-4d4e-ba48-a186b6d49a43\") " Oct 30 01:00:21.270164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.270130 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca" pod "641aa109-dc7d-4d4e-ba48-a186b6d49a43" (UID: "641aa109-dc7d-4d4e-ba48-a186b6d49a43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:21.270550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.270530 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641aa109-dc7d-4d4e-ba48-a186b6d49a43-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.271584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.271561 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm" pod "641aa109-dc7d-4d4e-ba48-a186b6d49a43" (UID: "641aa109-dc7d-4d4e-ba48-a186b6d49a43"). InnerVolumeSpecName "kube-api-access-b5cwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:21.280474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.280454 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:21.285601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.285584 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:21.371772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.371737 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\" (UniqueName: \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\") pod \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " Oct 30 01:00:21.371974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.371919 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\" (UniqueName: \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\") pod \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " Oct 30 01:00:21.372122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.372100 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\") pod \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\" (UID: \"16f41ef4-7167-4e1f-8c92-6b37b49df4fb\") " Oct 30 01:00:21.372210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.372178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\") pod \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\" (UID: \"72523efe-c57d-43c5-b44a-a3aa7edea5d7\") " Oct 30 01:00:21.372847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.372825 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cwm\" (UniqueName: \"kubernetes.io/projected/641aa109-dc7d-4d4e-ba48-a186b6d49a43-kube-api-access-b5cwm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.373164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.373145 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca" pod "72523efe-c57d-43c5-b44a-a3aa7edea5d7" (UID: "72523efe-c57d-43c5-b44a-a3aa7edea5d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:21.373442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.373422 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca" pod "16f41ef4-7167-4e1f-8c92-6b37b49df4fb" (UID: "16f41ef4-7167-4e1f-8c92-6b37b49df4fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:21.375109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.375075 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g" pod "16f41ef4-7167-4e1f-8c92-6b37b49df4fb" (UID: "16f41ef4-7167-4e1f-8c92-6b37b49df4fb"). InnerVolumeSpecName "kube-api-access-kzz4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:21.375215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.375198 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w" pod "72523efe-c57d-43c5-b44a-a3aa7edea5d7" (UID: "72523efe-c57d-43c5-b44a-a3aa7edea5d7"). InnerVolumeSpecName "kube-api-access-f2c4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:21.474409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.474299 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzz4g\" (UniqueName: \"kubernetes.io/projected/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-kube-api-access-kzz4g\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.474409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.474336 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f41ef4-7167-4e1f-8c92-6b37b49df4fb-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.474409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.474351 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72523efe-c57d-43c5-b44a-a3aa7edea5d7-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.474409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.474363 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2c4w\" (UniqueName: \"kubernetes.io/projected/72523efe-c57d-43c5-b44a-a3aa7edea5d7-kube-api-access-f2c4w\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:21.882851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.882820 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" path="/var/lib/kubelet/pods/37221d5b-9f20-4a94-bbc3-6e5837f058c9/volumes" Oct 30 01:00:21.883353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.883333 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" path="/var/lib/kubelet/pods/8a41fb40-5144-4e93-92b4-3dec7d41aab1/volumes" Oct 30 01:00:21.883751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.883736 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" path="/var/lib/kubelet/pods/9c87f67e-8633-4511-954e-9d38b1119b97/volumes" Oct 30 01:00:21.884559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.884540 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" path="/var/lib/kubelet/pods/d7b942c3-53a0-4482-a105-25303ffea6d7/volumes" Oct 30 01:00:21.969361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.969335 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" Oct 30 01:00:21.969886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.969456 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs" event={"ID":"72523efe-c57d-43c5-b44a-a3aa7edea5d7","Type":"ContainerDied","Data":"382c8f3e01310debed31a5b69a9fe7735411bb5e902e659d9c980867dc0afb68"} Oct 30 01:00:21.970153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.969970 5938 scope.go:122] "RemoveContainer" containerID="d085765a463a8780df0a2623b383b3fbd4137149d45f0598e15c8e7ee2057fa3" Oct 30 01:00:21.989361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:21.979335 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:22.020241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.013174 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" Oct 30 01:00:22.020241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.013629 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx" event={"ID":"16f41ef4-7167-4e1f-8c92-6b37b49df4fb","Type":"ContainerDied","Data":"b994a9518e54b4f6963780100d0d9a056202056e485e4a0a700bff07ee631f38"} Oct 30 01:00:22.028072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.027172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" event={"ID":"641aa109-dc7d-4d4e-ba48-a186b6d49a43","Type":"ContainerDied","Data":"918124ef7f969acea80b00ad5c185c5214acc371549b9941de34da70bc087e39"} Oct 30 01:00:22.028072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.027357 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p" Oct 30 01:00:22.032404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.032380 5938 scope.go:122] "RemoveContainer" containerID="caf9528c707bef79da682dbaa2188173fe298ea2050bcdbbf604f46237340f8d" Oct 30 01:00:22.081905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.072531 5938 scope.go:122] "RemoveContainer" containerID="4be0cf4b2864216ea25fbccc78a5a0c6c65a36d8d196e6f31e98858cd9469e11" Oct 30 01:00:22.141034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.140965 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:22.149787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.149756 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-cefb27-29888700-wvcxs"] Oct 30 01:00:22.220429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.220396 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:22.227394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.226063 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-021d6d-29888700-7g6lx"] Oct 30 01:00:22.278886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.275809 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:22.278886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.275890 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1f6ed-29888700-lws6p"] Oct 30 01:00:22.325664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.323706 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:22.402771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.402699 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:22.407739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.407716 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:22.413669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.413646 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\") pod \"a2ea8907-88e5-4304-8044-717f0f87493b\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " Oct 30 01:00:22.413834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.413819 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\" (UniqueName: \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\") pod \"a2ea8907-88e5-4304-8044-717f0f87493b\" (UID: \"a2ea8907-88e5-4304-8044-717f0f87493b\") " Oct 30 01:00:22.414004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.413980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca" pod "a2ea8907-88e5-4304-8044-717f0f87493b" (UID: "a2ea8907-88e5-4304-8044-717f0f87493b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:22.414570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.414550 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ea8907-88e5-4304-8044-717f0f87493b-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.415987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.415966 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j" pod "a2ea8907-88e5-4304-8044-717f0f87493b" (UID: "a2ea8907-88e5-4304-8044-717f0f87493b"). InnerVolumeSpecName "kube-api-access-rsh4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:22.515524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.515487 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\") pod \"79014303-b1c3-4d3a-970f-bee9997cc69c\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " Oct 30 01:00:22.515693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.515586 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\") pod \"da88969c-2f54-4e2e-be1f-9c2491b432c7\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " Oct 30 01:00:22.515693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.515688 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\" (UniqueName: \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\") pod \"79014303-b1c3-4d3a-970f-bee9997cc69c\" (UID: \"79014303-b1c3-4d3a-970f-bee9997cc69c\") " Oct 30 01:00:22.515798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.515734 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\" (UniqueName: \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\") pod \"da88969c-2f54-4e2e-be1f-9c2491b432c7\" (UID: \"da88969c-2f54-4e2e-be1f-9c2491b432c7\") " Oct 30 01:00:22.515891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.515821 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca" pod "79014303-b1c3-4d3a-970f-bee9997cc69c" (UID: "79014303-b1c3-4d3a-970f-bee9997cc69c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:22.516036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.516005 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca" pod "da88969c-2f54-4e2e-be1f-9c2491b432c7" (UID: "da88969c-2f54-4e2e-be1f-9c2491b432c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:22.516590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.516566 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh4j\" (UniqueName: \"kubernetes.io/projected/a2ea8907-88e5-4304-8044-717f0f87493b-kube-api-access-rsh4j\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.516590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.516592 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79014303-b1c3-4d3a-970f-bee9997cc69c-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.516756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.516602 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da88969c-2f54-4e2e-be1f-9c2491b432c7-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.517839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.517819 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg" pod "da88969c-2f54-4e2e-be1f-9c2491b432c7" (UID: "da88969c-2f54-4e2e-be1f-9c2491b432c7"). InnerVolumeSpecName "kube-api-access-qn8dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:22.517945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.517844 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv" pod "79014303-b1c3-4d3a-970f-bee9997cc69c" (UID: "79014303-b1c3-4d3a-970f-bee9997cc69c"). InnerVolumeSpecName "kube-api-access-xwfwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:22.618357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.618329 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwfwv\" (UniqueName: \"kubernetes.io/projected/79014303-b1c3-4d3a-970f-bee9997cc69c-kube-api-access-xwfwv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:22.618357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:22.618356 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8dg\" (UniqueName: \"kubernetes.io/projected/da88969c-2f54-4e2e-be1f-9c2491b432c7-kube-api-access-qn8dg\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.047910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.047847 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" Oct 30 01:00:23.048382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.047912 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v" event={"ID":"da88969c-2f54-4e2e-be1f-9c2491b432c7","Type":"ContainerDied","Data":"cb255bdca9e53be30409436c3e83193eefecd293001bbf93991042fc94eecf8a"} Oct 30 01:00:23.048382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.047984 5938 scope.go:122] "RemoveContainer" containerID="174f8ba987c17774e568a852c283d3c4e980da03465d3e8c9c8c05d27c9361b9" Oct 30 01:00:23.050563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.050532 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" event={"ID":"79014303-b1c3-4d3a-970f-bee9997cc69c","Type":"ContainerDied","Data":"8334fd9623297d27c0e20ef00f96ac7bc73eb61d9cd05cf3e0a977ade5ed1f2b"} Oct 30 01:00:23.050697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.050629 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm" Oct 30 01:00:23.058732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.058709 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" event={"ID":"a2ea8907-88e5-4304-8044-717f0f87493b","Type":"ContainerDied","Data":"6c29ea3c1f344dcb55f071398fbc86a579a5d861bcf23d020aa5b5550950c45f"} Oct 30 01:00:23.058856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.058803 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6" Oct 30 01:00:23.063701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.063681 5938 scope.go:122] "RemoveContainer" containerID="57fa3a30bb01a26c20cd4798bec7bb0514104bc146e55c347b9ce97d48e8d062" Oct 30 01:00:23.076613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.076595 5938 scope.go:122] "RemoveContainer" containerID="2493c10eefb6eb00620c731144029b78fb5074c5be69a2547cbf385e224fa75c" Oct 30 01:00:23.098832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.098806 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:23.110082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.110058 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8dbe73-29888700-k7x8v"] Oct 30 01:00:23.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.166779 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:23.180241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.180219 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-48df96-29888700-m87sm"] Oct 30 01:00:23.217538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.217460 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:23.224127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.224104 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-00e07a-29888700-zglx6"] Oct 30 01:00:23.879494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.879464 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" path="/var/lib/kubelet/pods/16f41ef4-7167-4e1f-8c92-6b37b49df4fb/volumes" Oct 30 01:00:23.879939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.879923 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" path="/var/lib/kubelet/pods/641aa109-dc7d-4d4e-ba48-a186b6d49a43/volumes" Oct 30 01:00:23.880359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.880346 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" path="/var/lib/kubelet/pods/72523efe-c57d-43c5-b44a-a3aa7edea5d7/volumes" Oct 30 01:00:23.881078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.881061 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" path="/var/lib/kubelet/pods/79014303-b1c3-4d3a-970f-bee9997cc69c/volumes" Oct 30 01:00:23.881488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.881474 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" path="/var/lib/kubelet/pods/a2ea8907-88e5-4304-8044-717f0f87493b/volumes" Oct 30 01:00:23.881894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.881878 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" path="/var/lib/kubelet/pods/da88969c-2f54-4e2e-be1f-9c2491b432c7/volumes" Oct 30 01:00:23.928084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:23.928048 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:25.615875 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:25.615826 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:27.395054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:27.395020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:28.840517 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:28.840476 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:00:29.275331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:29.275296 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:30.699899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:30.699857 5938 scope.go:122] "RemoveContainer" containerID="445f6d742f02c2529d74198abce0fe4001f87bc1d23d8bca31fd3031199930c1" Oct 30 01:00:30.710293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:30.710273 5938 scope.go:122] "RemoveContainer" containerID="44b84f6e8b035945b159b5de65be68ba80ddd055c3ef45373aa23612034c81b6" Oct 30 01:00:30.719773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:30.719754 5938 scope.go:122] "RemoveContainer" containerID="bf4e6c7eaeaf6ade6782071ad2f2a356634c8de825fed74cdc6c14bbfffa52af" Oct 30 01:00:30.998136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:30.998071 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:32.636218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:32.636190 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:34.522442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:34.522407 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:35.968089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:35.968059 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:36.623292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:36.623251 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:36.877903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:36.877817 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.051063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:37.051027 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.570572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:37.570535 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:38.441896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:38.441854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:38.897538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:38.897507 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:39.424788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:39.424727 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:40.627135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:40.627061 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:42.333859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:42.333824 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:43.860341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:43.860303 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:44.053348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:44.053314 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:44.192221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:44.192130 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 01:00:44.192437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:44.192412 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" containerID="cri-o://d676b1a80983773d0750bcf9fa89df0ca64e5987a9e20ffbc66ef4b98f49fce9" gracePeriod=30 Oct 30 01:00:44.281297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:44.281267 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:44.428308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:44.428260 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.137530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:45.137463 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.643282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:45.643242 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.834739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:45.834701 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.988941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:45.986929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:46.496086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:46.496045 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:48.547886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:48.545547 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:49.381010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.380656 5938 generic.go:356] "Generic (PLEG): container finished" podID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerID="d676b1a80983773d0750bcf9fa89df0ca64e5987a9e20ffbc66ef4b98f49fce9" exitCode=0 Oct 30 01:00:49.381010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.380828 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerDied","Data":"d676b1a80983773d0750bcf9fa89df0ca64e5987a9e20ffbc66ef4b98f49fce9"} Oct 30 01:00:49.381010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.380897 5938 scope.go:122] "RemoveContainer" containerID="613c85d3764687b299cea8049862bb9e561ef82d060b69c13a5773a112410193" Oct 30 01:00:49.658982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.658953 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 01:00:49.735235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735203 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735331 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735409 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735441 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735503 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735610 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735652 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:49.735760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735725 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735772 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\") pod \"874ef415-0d90-4025-a6ec-089a8cd4933d\" (UID: \"874ef415-0d90-4025-a6ec-089a8cd4933d\") " Oct 30 01:00:49.735950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735784 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:49.735950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735905 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:49.736130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.735943 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:49.736425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736373 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:49.736573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736553 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.736654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736575 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874ef415-0d90-4025-a6ec-089a8cd4933d-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.736654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736584 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.736654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736593 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.736654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.736601 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.738163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.738135 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:49.739353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.739322 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:00:49.739507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.739481 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "kube-api-access-8s8h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:49.739648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.739619 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo" pod "874ef415-0d90-4025-a6ec-089a8cd4933d" (UID: "874ef415-0d90-4025-a6ec-089a8cd4933d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:00:49.840170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.840141 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8h6\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-kube-api-access-8s8h6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.840170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.840167 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.840170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.840177 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:49.840492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:49.840185 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874ef415-0d90-4025-a6ec-089a8cd4933d-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:00:50.378933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.378897 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:50.444074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.444031 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" event={"ID":"874ef415-0d90-4025-a6ec-089a8cd4933d","Type":"ContainerDied","Data":"ed25be0cdafcbd72015b859949cb85448556ea560c496202ac5c227f7e27c37e"} Oct 30 01:00:50.444286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.444110 5938 scope.go:122] "RemoveContainer" containerID="d676b1a80983773d0750bcf9fa89df0ca64e5987a9e20ffbc66ef4b98f49fce9" Oct 30 01:00:50.444286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.444236 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp" Oct 30 01:00:50.517076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.517038 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 01:00:50.520305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.520268 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bab19a-openshift-default-649b8d548c-hj4gp"] Oct 30 01:00:50.874674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:50.874641 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.033810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:51.033765 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.880030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:51.879987 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" path="/var/lib/kubelet/pods/874ef415-0d90-4025-a6ec-089a8cd4933d/volumes" Oct 30 01:00:52.410890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:52.410356 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:52.550888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:52.550834 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:52.721284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:52.721200 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:52.917904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:52.917801 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:54.283386 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:54.283305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:55.958487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:55.958443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:57.777125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:57.777088 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:57.972834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:57.972752 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:58.524135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.524104 5938 generic.go:356] "Generic (PLEG): container finished" podID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerID="9c85deb20f58c7160245b6feebb9ae3b51daf9d611aaabda3a7a7cb82ecd120e" exitCode=0 Oct 30 01:00:58.524302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.524175 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerDied","Data":"9c85deb20f58c7160245b6feebb9ae3b51daf9d611aaabda3a7a7cb82ecd120e"} Oct 30 01:00:58.524571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.524557 5938 scope.go:122] "RemoveContainer" containerID="9c85deb20f58c7160245b6feebb9ae3b51daf9d611aaabda3a7a7cb82ecd120e" Oct 30 01:00:58.526394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.526364 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:58.526479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.526438 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:58.526479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:58.526470 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:00:58.838491 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:00:58.838412 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:00:59.535415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:59.535381 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerStarted","Data":"5546e736bc93a03bb769d91cf1fcaae39eab5f6953478c3382ee9da072ea45c5"} Oct 30 01:00:59.577362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:59.577335 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:59.801489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:59.801412 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:00:59.802478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:59.802451 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:00:59.802636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:00:59.802491 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:00.816858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.816816 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:00.817383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.816898 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:00.818690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.818664 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:01:00.819517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819497 5938 cpu_manager.go:414] "containerMap: removing container" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819516 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819550 5938 cpu_manager.go:414] "containerMap: removing container" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819558 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819570 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819577 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819584 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819588 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819597 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819602 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819609 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819613 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819621 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819626 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.819626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819632 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819640 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819645 5938 cpu_manager.go:414] "containerMap: removing container" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819650 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819657 5938 cpu_manager.go:414] "containerMap: removing container" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819661 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819669 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819673 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819682 5938 cpu_manager.go:414] "containerMap: removing container" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819686 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819692 5938 cpu_manager.go:414] "containerMap: removing container" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819696 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819703 5938 cpu_manager.go:414] "containerMap: removing container" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819707 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819720 5938 cpu_manager.go:414] "containerMap: removing container" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819724 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819731 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819735 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819741 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819746 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819752 5938 cpu_manager.go:414] "containerMap: removing container" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819757 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819767 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819772 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819784 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819790 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819798 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819802 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819823 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819829 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819838 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819843 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819854 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.819859 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820001 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820011 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820017 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820024 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ea8907-88e5-4304-8044-717f0f87493b" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820029 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820035 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820040 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820046 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820051 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820057 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820064 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820069 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820086 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820098 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820105 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820113 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820118 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820123 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820129 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820135 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b942c3-53a0-4482-a105-25303ffea6d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820141 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820151 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820156 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820164 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:00.820393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.820171 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:00.826184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.823592 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.850098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.850068 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:01:00.853202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.853172 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:00.855468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.855445 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-84b89f-openshift-default-dockercfg-x86x8\"" Oct 30 01:01:00.976199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.976154 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.976403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.976229 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhxv\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.976516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.976476 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.976742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.976711 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.981467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.976988 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.981467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.977073 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.981467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.977331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.981467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.977520 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:00.981467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:00.977662 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079289 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079318 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhxv\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079384 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079516 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079554 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079588 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079694 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.082968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.079770 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.099683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.099646 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.100058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.099699 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.100058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.100038 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.100290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.100268 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.101178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.101151 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.124962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.124493 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.126910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.126831 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.159979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.145192 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhxv\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.159979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.145675 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\") pod \"e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.159979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.153166 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:01.344439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.344347 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:01.515126 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:01:01.515097 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c9dda2_2849_413b_b7ee_04f30f19a310.slice/crio-27b4b4f2640a332e41512a4061a2b4e888fee6211cb15449b3247a4d66eb6c5a WatchSource:0}: Error finding container 27b4b4f2640a332e41512a4061a2b4e888fee6211cb15449b3247a4d66eb6c5a: Status 404 returned error can't find the container with id 27b4b4f2640a332e41512a4061a2b4e888fee6211cb15449b3247a4d66eb6c5a Oct 30 01:01:01.516433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.516407 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:01:01.518135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.518100 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:01.518228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.518167 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:01.518228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.518199 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:01.550930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.550849 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" event={"ID":"56c9dda2-2849-413b-b7ee-04f30f19a310","Type":"ContainerStarted","Data":"27b4b4f2640a332e41512a4061a2b4e888fee6211cb15449b3247a4d66eb6c5a"} Oct 30 01:01:01.806889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.806237 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:01.806889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:01.806302 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:02.560806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:02.560770 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" event={"ID":"56c9dda2-2849-413b-b7ee-04f30f19a310","Type":"ContainerStarted","Data":"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de"} Oct 30 01:01:02.630733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:02.630642 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podStartSLOduration=2.630624073 podStartE2EDuration="2.630624073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:02.63018803 +0000 UTC m=+1714.870423142" watchObservedRunningTime="2026-10-30 01:01:02.630624073 +0000 UTC m=+1714.870859185" Oct 30 01:01:02.808913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:02.804155 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:02.808913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:02.804224 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:03.026921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.026880 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:03.153693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.153658 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:03.155345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.155310 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:03.155518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.155391 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:03.805888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.803233 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:03.805888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:03.803298 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:04.154702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:04.154663 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:04.154932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:04.154723 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:04.662090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:04.662051 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:04.805889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:04.802226 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:04.805889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:04.802287 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:05.154051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:05.154015 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:05.154204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:05.154070 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:05.806890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:05.805292 5938 patch_prober.go:29] interesting pod/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:01:05.806890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:05.805357 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 01:01:06.154349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:06.154312 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:06.154535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:06.154362 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:06.452644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:06.452557 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:06.615260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:06.615218 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:01:06.814097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:06.814021 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:01:07.154656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:07.154624 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:07.154844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:07.154680 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:07.623645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:07.623618 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:01:08.154031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:08.154003 5938 patch_prober.go:29] interesting pod/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 01:01:08.154437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:08.154054 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 01:01:08.279364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:08.279312 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:09.204260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:09.204104 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:09.204760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:09.204640 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:09.207112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:09.207089 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:01:09.996241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:09.996180 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:11.687026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:11.686992 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:13.582817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:13.582759 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:15.268610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:15.268577 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:17.177065 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:17.175046 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:17.343885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:17.336553 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:18.722623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:18.722586 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:19.725656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:19.725620 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:21.728118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:21.728085 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:23.481237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:23.481170 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:24.124538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:24.124503 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:25.381535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:25.381492 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:27.042274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:27.042235 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:27.836410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:27.836377 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 01:01:27.868234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:27.868203 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 01:01:28.008961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:28.008603 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:28.150158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:28.150124 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:28.840555 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:01:28.840519 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:01:29.074562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:29.074526 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:30.815766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:30.815734 5938 scope.go:122] "RemoveContainer" containerID="db22a9bc52e0fab47c40607fe2738ac36560c0ffc3283d90a8d59bf5d8477c7e" Oct 30 01:01:30.880669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:30.880640 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:32.662602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:32.659643 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:32.837444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:32.837408 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" containerID="cri-o://4c25506101819f706dbe8750dae98514be11d68ac8f606c0542f4f0744273885" gracePeriod=25 Oct 30 01:01:32.868819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:32.868785 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" containerID="cri-o://9bec50c7c58ab67f256e97fe30db8d01726c1093b2c03583e5885e1f2300cd09" gracePeriod=25 Oct 30 01:01:34.203658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.203627 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:01:34.204616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204593 5938 cpu_manager.go:414] "containerMap: removing container" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204617 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874ef415-0d90-4025-a6ec-089a8cd4933d" containerName="istio-proxy" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204640 5938 cpu_manager.go:414] "containerMap: removing container" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204647 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204664 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204672 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204689 5938 cpu_manager.go:414] "containerMap: removing container" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204696 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204713 5938 cpu_manager.go:414] "containerMap: removing container" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:34.204738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204722 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204747 5938 cpu_manager.go:414] "containerMap: removing container" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204756 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da88969c-2f54-4e2e-be1f-9c2491b432c7" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204769 5938 cpu_manager.go:414] "containerMap: removing container" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204776 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204787 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.204794 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205055 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="72523efe-c57d-43c5-b44a-a3aa7edea5d7" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205070 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f41ef4-7167-4e1f-8c92-6b37b49df4fb" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205082 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="79014303-b1c3-4d3a-970f-bee9997cc69c" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205094 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c87f67e-8633-4511-954e-9d38b1119b97" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205104 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37221d5b-9f20-4a94-bbc3-6e5837f058c9" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205115 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="641aa109-dc7d-4d4e-ba48-a186b6d49a43" containerName="cleanup" Oct 30 01:01:34.205251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.205133 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a41fb40-5144-4e93-92b4-3dec7d41aab1" containerName="cleanup" Oct 30 01:01:34.209835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.209815 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.227098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.227075 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:01:34.234407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.234338 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ef1084-serving-cert\"" Oct 30 01:01:34.295936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.295899 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.296165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.296142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql42l\" (UniqueName: \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.296249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.296219 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.296310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.296300 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.328778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.328751 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.401537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.401491 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql42l\" (UniqueName: \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.412341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.412304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.412573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.412455 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.412707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.412673 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.413029 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:01:34.413003 5938 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-serving-cert: secret "maas-api-e2e-inf-a-ef1084-serving-cert" not found Oct 30 01:01:34.420998 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:01:34.420958 5938 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls podName:1099852e-0a22-42c5-a179-cce5afeeac7c nodeName:}" failed. No retries permitted until 2026-10-30 01:01:34.920924028 +0000 UTC m=+1747.161159136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls") pod "maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" (UID: "1099852e-0a22-42c5-a179-cce5afeeac7c") : secret "maas-api-e2e-inf-a-ef1084-serving-cert" not found Oct 30 01:01:34.428228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.427295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.434204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.434166 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.440532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.440502 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql42l\" (UniqueName: \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.494950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.494858 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.655645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.655612 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:34.934952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.934921 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:34.937326 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:34.937303 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:35.124629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:35.124470 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:35.442437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:35.442406 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:01:35.443715 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:01:35.443687 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1099852e_0a22_42c5_a179_cce5afeeac7c.slice/crio-e696de00ec3081016c018f570887b07b95768da35cf7835ba9f5ea2580b3d4d3 WatchSource:0}: Error finding container e696de00ec3081016c018f570887b07b95768da35cf7835ba9f5ea2580b3d4d3: Status 404 returned error can't find the container with id e696de00ec3081016c018f570887b07b95768da35cf7835ba9f5ea2580b3d4d3 Oct 30 01:01:35.866190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:35.866165 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" event={"ID":"1099852e-0a22-42c5-a179-cce5afeeac7c","Type":"ContainerStarted","Data":"e696de00ec3081016c018f570887b07b95768da35cf7835ba9f5ea2580b3d4d3"} Oct 30 01:01:36.282605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:36.282519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:36.877034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:36.876999 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" event={"ID":"1099852e-0a22-42c5-a179-cce5afeeac7c","Type":"ContainerStarted","Data":"cf389c09b00f6d6d1ef4007c60e97ae65a3b9802c7b03d9580e8d53fefc37010"} Oct 30 01:01:36.877430 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:36.877058 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:36.919044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:36.918995 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" podStartSLOduration=2.557977746 podStartE2EDuration="2.918981182s" totalImagesPullingTime="361.003436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:36.917259438 +0000 UTC m=+1749.157494557" watchObservedRunningTime="2026-10-30 01:01:36.918981182 +0000 UTC m=+1749.159216356" Oct 30 01:01:38.114745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:38.114712 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:38.571601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:38.571529 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:38.595391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:38.595360 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:01:39.603846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:39.603794 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:01:39.848169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:39.848133 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:40.107785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:40.107757 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:40.251556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:40.251519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:41.680416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:41.680378 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:42.890690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:42.890660 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:01:43.382196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:43.382165 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.160674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:45.160636 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.043964 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.042211 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.588596 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.588546 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.686629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.686593 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:01:47.691523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.691498 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.717022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.716991 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51b95e-openshift-default-dockercfg-vtc5s\"" Oct 30 01:01:47.721224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.721197 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:01:47.740484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.740450 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.827523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827490 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.827710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827579 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.827710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827666 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.827885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827774 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.827885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827842 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.827986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.827934 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.828058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.828038 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.828110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.828099 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.828239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.828220 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.929985 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930035 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930146 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930197 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930300 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930348 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930425 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930520 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930653 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.930817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.930792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.931116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.931093 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.932319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.932301 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.932676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.932657 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.952268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.952242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:47.952753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:47.952730 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\") pod \"e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:48.007176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.007149 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:48.329805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.329779 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:01:48.330294 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:01:48.330267 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb873cc77_195e_44f0_ac99_317f15ae2ee8.slice/crio-070d693508bf8897a6ffe863aa9dad4bd39dc0ca3b42d20e6336d95ec32dec82 WatchSource:0}: Error finding container 070d693508bf8897a6ffe863aa9dad4bd39dc0ca3b42d20e6336d95ec32dec82: Status 404 returned error can't find the container with id 070d693508bf8897a6ffe863aa9dad4bd39dc0ca3b42d20e6336d95ec32dec82 Oct 30 01:01:48.333060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.333027 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:48.333135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.333090 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:48.333135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.333119 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:01:48.344434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.344408 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 01:01:48.344663 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.344632 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" podUID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" containerName="maas-api" containerID="cri-o://de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51" gracePeriod=30 Oct 30 01:01:48.449108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.449067 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.657884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.650370 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:01:48.898885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.884319 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 01:01:48.946331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.945559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.996113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.995652 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\") pod \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " Oct 30 01:01:48.996113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.995932 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\") pod \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " Oct 30 01:01:48.996113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.995979 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\" (UniqueName: \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\") pod \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " Oct 30 01:01:48.996688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.996662 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\") pod \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\" (UID: \"7629ae34-a780-47a9-a2b5-753d54ab3fa6\") " Oct 30 01:01:48.999733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:48.999708 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca" pod "7629ae34-a780-47a9-a2b5-753d54ab3fa6" (UID: "7629ae34-a780-47a9-a2b5-753d54ab3fa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:49.007263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.007208 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs" pod "7629ae34-a780-47a9-a2b5-753d54ab3fa6" (UID: "7629ae34-a780-47a9-a2b5-753d54ab3fa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:49.008548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.008514 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls" pod "7629ae34-a780-47a9-a2b5-753d54ab3fa6" (UID: "7629ae34-a780-47a9-a2b5-753d54ab3fa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:01:49.013520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.013459 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl" pod "7629ae34-a780-47a9-a2b5-753d54ab3fa6" (UID: "7629ae34-a780-47a9-a2b5-753d54ab3fa6"). InnerVolumeSpecName "kube-api-access-t7rbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:49.022553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.022199 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" event={"ID":"b873cc77-195e-44f0-ac99-317f15ae2ee8","Type":"ContainerStarted","Data":"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02"} Oct 30 01:01:49.022553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.022260 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" event={"ID":"b873cc77-195e-44f0-ac99-317f15ae2ee8","Type":"ContainerStarted","Data":"070d693508bf8897a6ffe863aa9dad4bd39dc0ca3b42d20e6336d95ec32dec82"} Oct 30 01:01:49.024226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.024204 5938 generic.go:356] "Generic (PLEG): container finished" podID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" containerID="de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51" exitCode=0 Oct 30 01:01:49.024336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.024304 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" event={"ID":"7629ae34-a780-47a9-a2b5-753d54ab3fa6","Type":"ContainerDied","Data":"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51"} Oct 30 01:01:49.024403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.024342 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" event={"ID":"7629ae34-a780-47a9-a2b5-753d54ab3fa6","Type":"ContainerDied","Data":"61b80c51e8fd75765bb9a3f8bed713f6a013ad88c40de67700f135b6e461d416"} Oct 30 01:01:49.024403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.024371 5938 scope.go:122] "RemoveContainer" containerID="de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51" Oct 30 01:01:49.024561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.024547 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr" Oct 30 01:01:49.055743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.055719 5938 scope.go:122] "RemoveContainer" containerID="de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51" Oct 30 01:01:49.056162 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:01:49.056135 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51\": container with ID starting with de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51 not found: ID does not exist" containerID="de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51" Oct 30 01:01:49.056240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.056175 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51"} err="failed to get container status \"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51\": rpc error: code = NotFound desc = could not find container \"de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51\": container with ID starting with de4461203d6b235ee1ab9519d07bbfa9d6d27b6083575fcc3b01adcddccedb51 not found: ID does not exist" Oct 30 01:01:49.084482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.075930 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podStartSLOduration=2.075912749 podStartE2EDuration="2.075912749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:49.074237552 +0000 UTC m=+1761.314472666" watchObservedRunningTime="2026-10-30 01:01:49.075912749 +0000 UTC m=+1761.316147863" Oct 30 01:01:49.094995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.094956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.101370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.101340 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:49.101511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.101372 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rbl\" (UniqueName: \"kubernetes.io/projected/7629ae34-a780-47a9-a2b5-753d54ab3fa6-kube-api-access-t7rbl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:49.101511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.101389 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7629ae34-a780-47a9-a2b5-753d54ab3fa6-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:49.101511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.101404 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7629ae34-a780-47a9-a2b5-753d54ab3fa6-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:49.112646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.112624 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 01:01:49.116486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.116463 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6444d7-5cc6b684db-kdmdr"] Oct 30 01:01:49.237234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.237155 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.380800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.380768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.609467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.607183 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:01:49.748217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.748185 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.879289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:49.879208 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" path="/var/lib/kubelet/pods/7629ae34-a780-47a9-a2b5-753d54ab3fa6/volumes" Oct 30 01:01:50.009105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:50.009067 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:50.011685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:50.011652 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:50.011853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:50.011716 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:51.008913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:51.007962 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:51.008913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:51.008018 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:51.083091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:51.083055 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.009552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:52.009513 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:52.009971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:52.009579 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:52.801887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:52.801220 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:53.009493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.008989 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:53.009493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.009064 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:53.059156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.059069 5938 generic.go:356] "Generic (PLEG): container finished" podID="62df9578-ae21-4964-8f31-55a84641a57e" containerID="9bec50c7c58ab67f256e97fe30db8d01726c1093b2c03583e5885e1f2300cd09" exitCode=0 Oct 30 01:01:53.059156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.059145 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" event={"ID":"62df9578-ae21-4964-8f31-55a84641a57e","Type":"ContainerDied","Data":"9bec50c7c58ab67f256e97fe30db8d01726c1093b2c03583e5885e1f2300cd09"} Oct 30 01:01:53.061418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.061395 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerID="4c25506101819f706dbe8750dae98514be11d68ac8f606c0542f4f0744273885" exitCode=0 Oct 30 01:01:53.061623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.061474 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" event={"ID":"6e8e13af-9c87-4f40-9bee-3d97de22c836","Type":"ContainerDied","Data":"4c25506101819f706dbe8750dae98514be11d68ac8f606c0542f4f0744273885"} Oct 30 01:01:53.273280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.273249 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 01:01:53.321894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.321259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\") pod \"6e8e13af-9c87-4f40-9bee-3d97de22c836\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " Oct 30 01:01:53.321894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.321472 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\" (UniqueName: \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\") pod \"6e8e13af-9c87-4f40-9bee-3d97de22c836\" (UID: \"6e8e13af-9c87-4f40-9bee-3d97de22c836\") " Oct 30 01:01:53.322142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.322116 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume" pod "6e8e13af-9c87-4f40-9bee-3d97de22c836" (UID: "6e8e13af-9c87-4f40-9bee-3d97de22c836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:53.324473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.324444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v" pod "6e8e13af-9c87-4f40-9bee-3d97de22c836" (UID: "6e8e13af-9c87-4f40-9bee-3d97de22c836"). InnerVolumeSpecName "kube-api-access-49d5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:53.328129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.328110 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 01:01:53.424635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.424605 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\" (UniqueName: \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\") pod \"62df9578-ae21-4964-8f31-55a84641a57e\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " Oct 30 01:01:53.424832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.424708 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\") pod \"62df9578-ae21-4964-8f31-55a84641a57e\" (UID: \"62df9578-ae21-4964-8f31-55a84641a57e\") " Oct 30 01:01:53.425180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.425154 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume" pod "62df9578-ae21-4964-8f31-55a84641a57e" (UID: "62df9578-ae21-4964-8f31-55a84641a57e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:01:53.425741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.425668 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62df9578-ae21-4964-8f31-55a84641a57e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:53.425741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.425698 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49d5v\" (UniqueName: \"kubernetes.io/projected/6e8e13af-9c87-4f40-9bee-3d97de22c836-kube-api-access-49d5v\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:53.425741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.425712 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e8e13af-9c87-4f40-9bee-3d97de22c836-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:53.427728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.427689 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g" pod "62df9578-ae21-4964-8f31-55a84641a57e" (UID: "62df9578-ae21-4964-8f31-55a84641a57e"). InnerVolumeSpecName "kube-api-access-vzr7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:01:53.527614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:53.527583 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzr7g\" (UniqueName: \"kubernetes.io/projected/62df9578-ae21-4964-8f31-55a84641a57e-kube-api-access-vzr7g\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:01:54.009306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.009268 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:54.009515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.009327 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:54.071517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.071481 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" event={"ID":"62df9578-ae21-4964-8f31-55a84641a57e","Type":"ContainerDied","Data":"70fd9c479807fb772ad2217428d0978afed7f27d661c91c2dac9c4c2078288a2"} Oct 30 01:01:54.071517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.071499 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n" Oct 30 01:01:54.071999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.071549 5938 scope.go:122] "RemoveContainer" containerID="9bec50c7c58ab67f256e97fe30db8d01726c1093b2c03583e5885e1f2300cd09" Oct 30 01:01:54.081222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.081192 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" Oct 30 01:01:54.081362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.081252 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh" event={"ID":"6e8e13af-9c87-4f40-9bee-3d97de22c836","Type":"ContainerDied","Data":"b5cbe1c2434c84614f439ebacdc6b096c0f7585976d4b153ff191a2a7b4dedd2"} Oct 30 01:01:54.112066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.111993 5938 scope.go:122] "RemoveContainer" containerID="4c25506101819f706dbe8750dae98514be11d68ac8f606c0542f4f0744273885" Oct 30 01:01:54.162176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.162147 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 01:01:54.172454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.172422 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6444d7-86ddb5b999-ppd6n"] Oct 30 01:01:54.214717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.214653 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 01:01:54.218723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.218699 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6444d7-767fdb56d6-mc4fh"] Oct 30 01:01:54.526451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:54.526251 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.008472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:55.008440 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 01:01:55.008671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:55.008489 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 01:01:55.879102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:55.879073 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62df9578-ae21-4964-8f31-55a84641a57e" path="/var/lib/kubelet/pods/62df9578-ae21-4964-8f31-55a84641a57e/volumes" Oct 30 01:01:55.879553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:55.879517 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" path="/var/lib/kubelet/pods/6e8e13af-9c87-4f40-9bee-3d97de22c836/volumes" Oct 30 01:01:55.918734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:55.918703 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.013097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.013066 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:56.013540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.013526 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:56.014202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.014179 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:01:56.059039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.059011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.242229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.242148 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 01:01:56.242407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.242349 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" containerID="cri-o://26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38" gracePeriod=30 Oct 30 01:01:56.296790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.296729 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.456446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:56.456410 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:58.313063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:58.313031 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:58.837199 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:01:58.837155 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:01:59.973802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:01:59.973766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:00.113514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:00.113479 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:01.531012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.530955 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 01:02:01.647630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647597 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.647818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647800 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.647904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647851 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.647974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647912 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.647974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647961 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.648093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.647988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.648093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648020 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.648093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648023 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:01.648260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648144 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:01.648260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648072 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.648362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648292 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:01.648419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648386 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\") pod \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\" (UID: \"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d\") " Oct 30 01:02:01.649005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.648950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:01.649366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.649345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:01.650180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650151 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.650180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650182 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.650363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650195 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.650363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650207 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.650363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650222 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.650937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650825 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:02:01.650937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.650831 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:01.651267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.651245 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "kube-api-access-lllcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:01.651332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.651284 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy" pod "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" (UID: "e4d5ec8a-460c-4bdd-a361-5cb14bd4537d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:01.751390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.751367 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.751390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.751391 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.751569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.751401 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lllcp\" (UniqueName: \"kubernetes.io/projected/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-kube-api-access-lllcp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.751569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.751410 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:01.760095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:01.760074 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:02.156450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.156370 5938 generic.go:356] "Generic (PLEG): container finished" podID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerID="26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38" exitCode=0 Oct 30 01:02:02.156450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.156417 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" Oct 30 01:02:02.156675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.156451 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" event={"ID":"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d","Type":"ContainerDied","Data":"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38"} Oct 30 01:02:02.156675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.156503 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw" event={"ID":"e4d5ec8a-460c-4bdd-a361-5cb14bd4537d","Type":"ContainerDied","Data":"faef4b6534c01495391ce980888dc0ab3becd790a6bc56afdcc42ad62fcb7633"} Oct 30 01:02:02.156675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.156526 5938 scope.go:122] "RemoveContainer" containerID="26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38" Oct 30 01:02:02.167543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.167528 5938 scope.go:122] "RemoveContainer" containerID="26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38" Oct 30 01:02:02.167811 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:02.167785 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38\": container with ID starting with 26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38 not found: ID does not exist" containerID="26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38" Oct 30 01:02:02.167855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.167820 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38"} err="failed to get container status \"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38\": rpc error: code = NotFound desc = could not find container \"26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38\": container with ID starting with 26476c4d184a545d6831f9107795ec1e38ebb04a9d6165c6788a6b4e9a985b38 not found: ID does not exist" Oct 30 01:02:02.219937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.219913 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 01:02:02.231273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:02.231251 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6444d7-openshift-default-7bf76bdcd9-qtjxw"] Oct 30 01:02:03.457572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:03.457544 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:03.878657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:03.878619 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" path="/var/lib/kubelet/pods/e4d5ec8a-460c-4bdd-a361-5cb14bd4537d/volumes" Oct 30 01:02:05.108477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:05.108334 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:06.862510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:06.862472 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:07.434096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:07.434059 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 01:02:07.467410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:07.467373 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 01:02:07.594045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:07.593989 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:07.738464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:07.738347 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.779854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:08.779819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:10.083963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.083920 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.140995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.140948 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.178984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.178940 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.270389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.270346 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.308050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.307079 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.383847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.382970 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:10.394971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:10.394933 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:11.269055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:11.269020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:12.435223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:12.435187 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" containerID="cri-o://2f61f03ab94ab3fe973e46d151872a2ad25e1ad55e2553101c063f6ca7016eec" gracePeriod=25 Oct 30 01:02:12.467746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:12.467709 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" containerID="cri-o://6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece" gracePeriod=25 Oct 30 01:02:12.994479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:12.994425 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:14.630023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:14.629986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:16.519105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:16.519069 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:18.216345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:18.216297 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:20.071984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:20.071955 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:20.196685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:20.196650 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:20.196856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:20.196699 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:21.888912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:21.888846 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:23.121796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:23.121726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:23.575515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:23.575416 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:23.772127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:23.772097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:24.034231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.034202 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:02:24.035026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035009 5938 cpu_manager.go:414] "containerMap: removing container" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" Oct 30 01:02:24.035075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035027 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" Oct 30 01:02:24.035075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035065 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" Oct 30 01:02:24.035075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035071 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" Oct 30 01:02:24.035181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035084 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" containerName="maas-api" Oct 30 01:02:24.035181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035089 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" containerName="maas-api" Oct 30 01:02:24.035181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035100 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" Oct 30 01:02:24.035181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035104 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" Oct 30 01:02:24.035315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035218 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="62df9578-ae21-4964-8f31-55a84641a57e" containerName="payload-pre-processing" Oct 30 01:02:24.035315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035235 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d5ec8a-460c-4bdd-a361-5cb14bd4537d" containerName="istio-proxy" Oct 30 01:02:24.035315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035246 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7629ae34-a780-47a9-a2b5-753d54ab3fa6" containerName="maas-api" Oct 30 01:02:24.035315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.035260 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8e13af-9c87-4f40-9bee-3d97de22c836" containerName="payload-processing" Oct 30 01:02:24.039384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.039363 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.063395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.063366 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:02:24.071639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.071616 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19bc68-openshift-default-dockercfg-vd65x\"" Oct 30 01:02:24.128682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.128617 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:24.146400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146376 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146406 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146525 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146562 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146634 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146655 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146678 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4mw\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146808 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.146955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.146835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.251725 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.251788 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.251917 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.251950 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.251982 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4mw\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.252154 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.252189 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.252361 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.252989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.252392 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.254417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.254390 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.255229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.255204 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.255636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.255616 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.277883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.256474 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.278129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.260736 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.278296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.260991 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.285925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.280984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:24.322532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.320761 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.322532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.321046 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.336487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.336458 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4mw\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\") pod \"e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.369134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.368855 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:24.840241 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:02:24.840213 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d49da6c_d1a1_43c5_8b3c_65f638d567e8.slice/crio-ae3dde5f641ed75b45e150c048b196c5f6207fed3967687920de16c4a60f17f2 WatchSource:0}: Error finding container ae3dde5f641ed75b45e150c048b196c5f6207fed3967687920de16c4a60f17f2: Status 404 returned error can't find the container with id ae3dde5f641ed75b45e150c048b196c5f6207fed3967687920de16c4a60f17f2 Oct 30 01:02:24.842992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.842954 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:24.843090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.843040 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:24.843090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.843074 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:24.850324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:24.850299 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:02:25.443643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:25.443598 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:25.502577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:25.502539 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerStarted","Data":"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5"} Oct 30 01:02:25.502760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:25.502622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerStarted","Data":"ae3dde5f641ed75b45e150c048b196c5f6207fed3967687920de16c4a60f17f2"} Oct 30 01:02:25.566169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:25.566121 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podStartSLOduration=1.566106738 podStartE2EDuration="1.566106738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:25.562786455 +0000 UTC m=+1797.803021564" watchObservedRunningTime="2026-10-30 01:02:25.566106738 +0000 UTC m=+1797.806341896" Oct 30 01:02:26.370413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:26.370372 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:26.376645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:26.376621 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:26.511066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:26.511034 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:26.512039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:26.512015 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:02:27.287256 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:27.287225 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:27.428089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:27.428050 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:28.227986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:28.227946 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:28.385555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:28.385308 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:28.838533 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:28.838481 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:02:29.196809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:29.196659 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:29.892064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:29.892035 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:30.067210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.067178 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:30.200852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.200763 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:30.204352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.204322 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:02:30.456687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.456599 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:02:30.477630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.477607 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:02:30.848353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:30.848281 5938 scope.go:122] "RemoveContainer" containerID="a49c298d1bb73f806330ca6dbedfca87cb9675ea0c8a5b935bda336a9a1f1338" Oct 30 01:02:31.227683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:31.227654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:32.560366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.560337 5938 generic.go:356] "Generic (PLEG): container finished" podID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerID="2f61f03ab94ab3fe973e46d151872a2ad25e1ad55e2553101c063f6ca7016eec" exitCode=0 Oct 30 01:02:32.560717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.560407 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" event={"ID":"1fae86f4-27c3-463e-9ec7-d3884a64c46f","Type":"ContainerDied","Data":"2f61f03ab94ab3fe973e46d151872a2ad25e1ad55e2553101c063f6ca7016eec"} Oct 30 01:02:32.775247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.775217 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 01:02:32.805353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.805327 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 01:02:32.931990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.931957 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\" (UniqueName: \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\") pod \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " Oct 30 01:02:32.932193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.932063 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\") pod \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " Oct 30 01:02:32.932193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.932166 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\" (UniqueName: \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\") pod \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\" (UID: \"2f92dfc8-9e80-479a-b4de-05e0e183ea24\") " Oct 30 01:02:32.932348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.932323 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\") pod \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\" (UID: \"1fae86f4-27c3-463e-9ec7-d3884a64c46f\") " Oct 30 01:02:32.932478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.932450 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume" pod "2f92dfc8-9e80-479a-b4de-05e0e183ea24" (UID: "2f92dfc8-9e80-479a-b4de-05e0e183ea24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:32.932972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.932936 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume" pod "1fae86f4-27c3-463e-9ec7-d3884a64c46f" (UID: "1fae86f4-27c3-463e-9ec7-d3884a64c46f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:32.933211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.933185 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fae86f4-27c3-463e-9ec7-d3884a64c46f-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:32.933270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.933212 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f92dfc8-9e80-479a-b4de-05e0e183ea24-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:32.934451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.934428 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w" pod "2f92dfc8-9e80-479a-b4de-05e0e183ea24" (UID: "2f92dfc8-9e80-479a-b4de-05e0e183ea24"). InnerVolumeSpecName "kube-api-access-25s9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:32.934508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:32.934445 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg" pod "1fae86f4-27c3-463e-9ec7-d3884a64c46f" (UID: "1fae86f4-27c3-463e-9ec7-d3884a64c46f"). InnerVolumeSpecName "kube-api-access-qrnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:33.034198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.034137 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25s9w\" (UniqueName: \"kubernetes.io/projected/2f92dfc8-9e80-479a-b4de-05e0e183ea24-kube-api-access-25s9w\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:33.034198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.034160 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnvg\" (UniqueName: \"kubernetes.io/projected/1fae86f4-27c3-463e-9ec7-d3884a64c46f-kube-api-access-qrnvg\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:33.071199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.066619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:33.573295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.573256 5938 generic.go:356] "Generic (PLEG): container finished" podID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerID="6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece" exitCode=0 Oct 30 01:02:33.573777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.573310 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" Oct 30 01:02:33.573777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.573335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" event={"ID":"2f92dfc8-9e80-479a-b4de-05e0e183ea24","Type":"ContainerDied","Data":"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece"} Oct 30 01:02:33.573777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.573389 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj" event={"ID":"2f92dfc8-9e80-479a-b4de-05e0e183ea24","Type":"ContainerDied","Data":"177f59b3eb3edddb19ba04cd39fd298a718b1f3f15b0153c3781758f28e0557b"} Oct 30 01:02:33.573777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.573420 5938 scope.go:122] "RemoveContainer" containerID="6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece" Oct 30 01:02:33.575122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.575096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" event={"ID":"1fae86f4-27c3-463e-9ec7-d3884a64c46f","Type":"ContainerDied","Data":"e9de2f2b27ec165d68642970bfe8977a81de7f27753cae40d79c2d4aff5c8b58"} Oct 30 01:02:33.575244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.575136 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt" Oct 30 01:02:33.589299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.589059 5938 scope.go:122] "RemoveContainer" containerID="6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece" Oct 30 01:02:33.591543 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:33.591503 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece\": container with ID starting with 6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece not found: ID does not exist" containerID="6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece" Oct 30 01:02:33.591613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.591546 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece"} err="failed to get container status \"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece\": rpc error: code = NotFound desc = could not find container \"6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece\": container with ID starting with 6d5df269199fd87e3c5b618b9a4cd876dfdde48395f2b048c35d62979d641ece not found: ID does not exist" Oct 30 01:02:33.591613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.591577 5938 scope.go:122] "RemoveContainer" containerID="2f61f03ab94ab3fe973e46d151872a2ad25e1ad55e2553101c063f6ca7016eec" Oct 30 01:02:33.646647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.644750 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 01:02:33.652916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.651141 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b3bdf9-6976bd6ff8-sn6nt"] Oct 30 01:02:33.679713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.679683 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 01:02:33.683920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.683892 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b3bdf9-867787698c-vlrxj"] Oct 30 01:02:33.881140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.881111 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" path="/var/lib/kubelet/pods/1fae86f4-27c3-463e-9ec7-d3884a64c46f/volumes" Oct 30 01:02:33.881553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:33.881536 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" path="/var/lib/kubelet/pods/2f92dfc8-9e80-479a-b4de-05e0e183ea24/volumes" Oct 30 01:02:34.755909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:34.755850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:36.448805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:36.448776 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:38.225841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:38.225803 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:38.749220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:38.749186 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:39.448888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:39.448325 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:39.602285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:39.602219 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:39.605880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:39.605838 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 01:02:39.606124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:39.606100 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" podUID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" containerName="maas-api" containerID="cri-o://f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea" gracePeriod=30 Oct 30 01:02:39.822489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:39.822391 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:40.058748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.058673 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 01:02:40.165884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.165467 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") pod \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " Oct 30 01:02:40.165884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.165566 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\") pod \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " Oct 30 01:02:40.165884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.165692 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\" (UniqueName: \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\") pod \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " Oct 30 01:02:40.165884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.165814 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\") pod \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\" (UID: \"2a785cc9-3c64-4ddf-91c8-0b75172f13d7\") " Oct 30 01:02:40.188910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.171603 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca" pod "2a785cc9-3c64-4ddf-91c8-0b75172f13d7" (UID: "2a785cc9-3c64-4ddf-91c8-0b75172f13d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:40.199842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.197365 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv" pod "2a785cc9-3c64-4ddf-91c8-0b75172f13d7" (UID: "2a785cc9-3c64-4ddf-91c8-0b75172f13d7"). InnerVolumeSpecName "kube-api-access-v84lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:40.199842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.198077 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls" pod "2a785cc9-3c64-4ddf-91c8-0b75172f13d7" (UID: "2a785cc9-3c64-4ddf-91c8-0b75172f13d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:40.232947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.205233 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs" pod "2a785cc9-3c64-4ddf-91c8-0b75172f13d7" (UID: "2a785cc9-3c64-4ddf-91c8-0b75172f13d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:02:40.288825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.284462 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:40.288825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.284500 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84lv\" (UniqueName: \"kubernetes.io/projected/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-kube-api-access-v84lv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:40.288825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.284515 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:40.288825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.284528 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a785cc9-3c64-4ddf-91c8-0b75172f13d7-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:40.649902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.649854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:40.659198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.659166 5938 generic.go:356] "Generic (PLEG): container finished" podID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" containerID="f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea" exitCode=0 Oct 30 01:02:40.659417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.659249 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" Oct 30 01:02:40.659517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.659444 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" event={"ID":"2a785cc9-3c64-4ddf-91c8-0b75172f13d7","Type":"ContainerDied","Data":"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea"} Oct 30 01:02:40.659517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.659491 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw" event={"ID":"2a785cc9-3c64-4ddf-91c8-0b75172f13d7","Type":"ContainerDied","Data":"4cdf765f4ef6a1286fb68e541a7994858b607971f960b9663ca564eabcc605ba"} Oct 30 01:02:40.659643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.659523 5938 scope.go:122] "RemoveContainer" containerID="f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea" Oct 30 01:02:40.671600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.671583 5938 scope.go:122] "RemoveContainer" containerID="f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea" Oct 30 01:02:40.671835 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:40.671807 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea\": container with ID starting with f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea not found: ID does not exist" containerID="f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea" Oct 30 01:02:40.671978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.671843 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea"} err="failed to get container status \"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea\": rpc error: code = NotFound desc = could not find container \"f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea\": container with ID starting with f390941a073ba2d43d05810063b3784a60a9c603c15989bdf451b368e86b7eea not found: ID does not exist" Oct 30 01:02:40.713960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.713939 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 01:02:40.717639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.717615 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b3bdf9-6548487c94-fxdcw"] Oct 30 01:02:40.794109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:40.794077 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:41.350888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:41.350462 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:41.879387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:41.879356 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" path="/var/lib/kubelet/pods/2a785cc9-3c64-4ddf-91c8-0b75172f13d7/volumes" Oct 30 01:02:42.752538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:42.752480 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:44.427238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.427184 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:44.839191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839120 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:02:44.839846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839831 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" Oct 30 01:02:44.839908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839847 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" Oct 30 01:02:44.839908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839890 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" Oct 30 01:02:44.839908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839895 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" Oct 30 01:02:44.840006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839926 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" containerName="maas-api" Oct 30 01:02:44.840006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.839931 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" containerName="maas-api" Oct 30 01:02:44.840071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.840060 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fae86f4-27c3-463e-9ec7-d3884a64c46f" containerName="payload-processing" Oct 30 01:02:44.840105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.840072 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f92dfc8-9e80-479a-b4de-05e0e183ea24" containerName="payload-pre-processing" Oct 30 01:02:44.840105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.840085 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a785cc9-3c64-4ddf-91c8-0b75172f13d7" containerName="maas-api" Oct 30 01:02:44.843552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.843530 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.857767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.857745 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1f6ed-openshift-default-dockercfg-wjgfj\"" Oct 30 01:02:44.873840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.873807 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:02:44.988013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.987981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988026 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988256 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988292 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988354 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swbt\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988421 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:44.988519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:44.988499 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089500 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089574 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swbt\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089678 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.089779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089702 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089844 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.089916 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090010 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090053 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090241 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090498 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090555 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.090839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.090813 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.092025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.092000 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.092148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.092126 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.112028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.111998 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.112207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.112186 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swbt\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.157283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.157261 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:45.403821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.403719 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:45.735956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.735923 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:02:45.736940 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:02:45.736908 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dc49ed_7fe2_450a_9b07_7c1e2ae56567.slice/crio-0aaec36f3829aaf701edb971c1ee49bd4aeed586bbd561642079befdf64a2ba0 WatchSource:0}: Error finding container 0aaec36f3829aaf701edb971c1ee49bd4aeed586bbd561642079befdf64a2ba0: Status 404 returned error can't find the container with id 0aaec36f3829aaf701edb971c1ee49bd4aeed586bbd561642079befdf64a2ba0 Oct 30 01:02:45.739941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.739905 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:45.740078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.739976 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:45.740078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:45.740008 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:46.373873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:46.373821 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:46.708355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:46.708272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" event={"ID":"01dc49ed-7fe2-450a-9b07-7c1e2ae56567","Type":"ContainerStarted","Data":"2db03c89ab30a2d3406697b7d3a22034fa52d7d3547219d8396fc7d032f7cf9d"} Oct 30 01:02:46.708355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:46.708333 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" event={"ID":"01dc49ed-7fe2-450a-9b07-7c1e2ae56567","Type":"ContainerStarted","Data":"0aaec36f3829aaf701edb971c1ee49bd4aeed586bbd561642079befdf64a2ba0"} Oct 30 01:02:46.769002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:46.768952 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podStartSLOduration=2.768938814 podStartE2EDuration="2.768938814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:46.767223062 +0000 UTC m=+1819.007458177" watchObservedRunningTime="2026-10-30 01:02:46.768938814 +0000 UTC m=+1819.009173918" Oct 30 01:02:47.157441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:47.157407 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:47.158936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:47.158911 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:47.159000 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:47.158976 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:47.825693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:47.825661 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:47.970517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:47.970482 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.158484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.158448 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:48.158696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.158505 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:48.158973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.158949 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 01:02:48.159236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.159211 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" containerID="cri-o://f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53" gracePeriod=30 Oct 30 01:02:48.235358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.235323 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.373076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:48.373040 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.121093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.121043 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.157823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.157788 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:49.158028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.157839 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:49.538691 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.538656 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:02:49.542483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.542462 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.565270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.557763 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-beeba4-openshift-default-dockercfg-5vtrz\"" Oct 30 01:02:49.576926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.576894 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.576943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.576981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577099 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577117 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:02:49.577345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577321 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577440 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577474 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vfb8\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.577582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.577556 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.636440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.636397 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.679074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679036 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679137 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679225 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679250 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfb8\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679302 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679327 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679407 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679457 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679506 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679771 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.679936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.679911 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.680127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.680104 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.680230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.680213 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.682195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.682049 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.682497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.682477 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.745609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.740672 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfb8\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.846941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.797553 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:49.846941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.807560 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.892887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:49.872896 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:50.064807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.058885 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:50.158417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.158385 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:50.159137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.159103 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:50.480059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.480015 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:02:50.509775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.509732 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:50.510039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.510005 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:50.510168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.510052 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:02:50.858027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.857992 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerStarted","Data":"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b"} Oct 30 01:02:50.858196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.858056 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerStarted","Data":"a43120678ce4bf3c0a67a44602b5f727ac9b9eabffb95bd821c2a16d4449beef"} Oct 30 01:02:50.901882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.899361 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:50.917604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:50.917545 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podStartSLOduration=1.91752265 podStartE2EDuration="1.91752265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:50.901229716 +0000 UTC m=+1823.141464830" watchObservedRunningTime="2026-10-30 01:02:50.91752265 +0000 UTC m=+1823.157757765" Oct 30 01:02:51.158036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:51.157951 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:51.158266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:51.158240 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:51.775263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:51.775228 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:51.910137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:51.910103 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:52.158281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:52.158253 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:02:52.158437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:52.158309 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 01:02:52.876600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:52.876565 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:52.877656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:52.877633 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:02:53.234571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.234539 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:53.234927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.234905 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:53.235713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.235694 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:02:53.426385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.423656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.518918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.518889 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 01:02:53.569452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.569415 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.638174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638195 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638311 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638387 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638432 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638468 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638537 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:53.638669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638548 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.638787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638744 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:53.638939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638781 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:53.638939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638927 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\") pod \"254296d8-2085-4cc9-b2dd-63d78b0643c8\" (UID: \"254296d8-2085-4cc9-b2dd-63d78b0643c8\") " Oct 30 01:02:53.639059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.638931 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:53.639415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:02:53.639836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639815 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.639967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639843 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.639967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639859 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.639967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639890 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254296d8-2085-4cc9-b2dd-63d78b0643c8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.639967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.639904 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.641190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.641117 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:02:53.641422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.641401 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:02:53.641509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.641429 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:53.641999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.641982 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp" pod "254296d8-2085-4cc9-b2dd-63d78b0643c8" (UID: "254296d8-2085-4cc9-b2dd-63d78b0643c8"). InnerVolumeSpecName "kube-api-access-8k5kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:02:53.716393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.716305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.741904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.741850 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.741904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.741904 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.742094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.741917 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254296d8-2085-4cc9-b2dd-63d78b0643c8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.742094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.741929 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5kp\" (UniqueName: \"kubernetes.io/projected/254296d8-2085-4cc9-b2dd-63d78b0643c8-kube-api-access-8k5kp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:02:53.885965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.885931 5938 generic.go:356] "Generic (PLEG): container finished" podID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerID="f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53" exitCode=0 Oct 30 01:02:53.886455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.885977 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" Oct 30 01:02:53.886455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.886006 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerDied","Data":"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53"} Oct 30 01:02:53.886455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.886060 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7" event={"ID":"254296d8-2085-4cc9-b2dd-63d78b0643c8","Type":"ContainerDied","Data":"27d76376d9b4e59e176de84ac1f458f87b4aaeff383b58185099a2df225417e1"} Oct 30 01:02:53.886455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.886091 5938 scope.go:122] "RemoveContainer" containerID="f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53" Oct 30 01:02:53.902371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.902344 5938 scope.go:122] "RemoveContainer" containerID="54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753" Oct 30 01:02:53.912472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.912457 5938 scope.go:122] "RemoveContainer" containerID="f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53" Oct 30 01:02:53.912743 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:53.912720 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53\": container with ID starting with f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53 not found: ID does not exist" containerID="f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53" Oct 30 01:02:53.912797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.912753 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53"} err="failed to get container status \"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53\": rpc error: code = NotFound desc = could not find container \"f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53\": container with ID starting with f468de90676707885700e167aa57805ad42331f4f6f72d4419233fde15823e53 not found: ID does not exist" Oct 30 01:02:53.912797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.912770 5938 scope.go:122] "RemoveContainer" containerID="54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753" Oct 30 01:02:53.913064 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:53.913047 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753\": container with ID starting with 54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753 not found: ID does not exist" containerID="54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753" Oct 30 01:02:53.913112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.913073 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753"} err="failed to get container status \"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753\": rpc error: code = NotFound desc = could not find container \"54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753\": container with ID starting with 54102e774551ebc6a696fafb2c4b0ce3440b18bccdd5a919d337fb0c2cfc0753 not found: ID does not exist" Oct 30 01:02:53.941875 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.941839 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 01:02:53.953020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:53.952997 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b3bdf9-openshift-default-755bccddf-cz2h7"] Oct 30 01:02:55.566069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.566036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.812051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.812018 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.880118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.880084 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" path="/var/lib/kubelet/pods/254296d8-2085-4cc9-b2dd-63d78b0643c8/volumes" Oct 30 01:02:55.882626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.882603 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:02:55.883440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.883420 5938 cpu_manager.go:414] "containerMap: removing container" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.883567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.883440 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.883567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.883469 5938 cpu_manager.go:414] "containerMap: removing container" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.883567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.883476 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.883724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.883666 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.888915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.888896 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:55.906775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.906754 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:02:55.913393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.913372 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-beeba4\"" Oct 30 01:02:55.964619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.964587 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:02:55.966082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.966056 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="254296d8-2085-4cc9-b2dd-63d78b0643c8" containerName="istio-proxy" Oct 30 01:02:55.968506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.968480 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.970636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.970605 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:55.974467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.974439 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbcl\" (UniqueName: \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:55.974850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.974654 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:55.986183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.986159 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-beeba4-dockercfg-4k9m8\"" Oct 30 01:02:55.987029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:55.987006 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:02:56.110796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.110747 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmd7\" (UniqueName: \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.111028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.110997 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.111128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.111105 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbcl\" (UniqueName: \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.111248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.111227 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.112350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.112317 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.244905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.243311 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmd7\" (UniqueName: \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.244905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.243553 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.245358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.245295 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.263638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.263593 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbcl\" (UniqueName: \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\") pod \"payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.266087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.266061 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmd7\" (UniqueName: \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\") pod \"payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.290638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.290544 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.504657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.504588 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.607480 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:02:56.607449 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46574bef_398f_4d7e_bbda_0eecd6ce2f31.slice/crio-d39e9f6ae3d81f0e74e781a639e6220287e9766b5b4ef029db8e7e340b2d34fc WatchSource:0}: Error finding container d39e9f6ae3d81f0e74e781a639e6220287e9766b5b4ef029db8e7e340b2d34fc: Status 404 returned error can't find the container with id d39e9f6ae3d81f0e74e781a639e6220287e9766b5b4ef029db8e7e340b2d34fc Oct 30 01:02:56.607904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.607848 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:02:56.811011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.810982 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:02:56.912194 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.912168 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" event={"ID":"c0197d7f-b6ee-4137-8efe-010ecea1cbf2","Type":"ContainerStarted","Data":"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a"} Oct 30 01:02:56.912319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.912218 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" event={"ID":"c0197d7f-b6ee-4137-8efe-010ecea1cbf2","Type":"ContainerStarted","Data":"eeeabe9ddf0a1e53f7e54cb447166db16baf79d1517eb19db0892aaeccba5def"} Oct 30 01:02:56.912319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.912245 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:02:56.913915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.913889 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" event={"ID":"46574bef-398f-4d7e-bbda-0eecd6ce2f31","Type":"ContainerStarted","Data":"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c"} Oct 30 01:02:56.913993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.913935 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" event={"ID":"46574bef-398f-4d7e-bbda-0eecd6ce2f31","Type":"ContainerStarted","Data":"d39e9f6ae3d81f0e74e781a639e6220287e9766b5b4ef029db8e7e340b2d34fc"} Oct 30 01:02:56.913993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.913968 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:02:56.953100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.953055 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" podStartSLOduration=1.9530422509999998 podStartE2EDuration="1.953042251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:56.949974878 +0000 UTC m=+1829.190209992" watchObservedRunningTime="2026-10-30 01:02:56.953042251 +0000 UTC m=+1829.193277364" Oct 30 01:02:56.990139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:56.990096 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" podStartSLOduration=1.990083173 podStartE2EDuration="1.990083173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:56.988049443 +0000 UTC m=+1829.228284558" watchObservedRunningTime="2026-10-30 01:02:56.990083173 +0000 UTC m=+1829.230318286" Oct 30 01:02:57.578969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:57.578936 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.837843 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:02:58.837807 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:02:59.203680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:02:59.203648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.893297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:00.893260 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:02.642985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:02.642952 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:04.256063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:04.256035 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:05.875075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:05.875035 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.671729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:07.671697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.924952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:07.924872 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:03:07.924952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:07.924934 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:03:09.458993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:09.458959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:09.604985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:09.604949 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:11.310892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:11.309215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:13.181500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:13.181443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:14.870812 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:14.870777 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:15.149340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:15.149256 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:15.663526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:15.663488 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:16.480690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:16.480656 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 01:03:16.481133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:16.480842 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" containerID="cri-o://2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318" gracePeriod=30 Oct 30 01:03:16.932055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:16.931962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:18.778402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:18.778365 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:20.535034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:20.534994 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:21.758521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.758466 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 01:03:21.851058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851030 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851092 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851161 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851197 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851293 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:21.851413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851348 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851421 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:21.851534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851430 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851501 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:21.851534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851529 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851586 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.851770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851633 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\") pod \"4e630456-41c9-408e-b99c-89cd312e9b78\" (UID: \"4e630456-41c9-408e-b99c-89cd312e9b78\") " Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851880 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.851908 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.852388 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e630456-41c9-408e-b99c-89cd312e9b78-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.852413 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.852428 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.852443 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.852512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.852459 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.853835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.853808 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:21.853981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.853917 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "kube-api-access-fqfbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:21.854066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.854050 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:21.854167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.854148 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token" pod "4e630456-41c9-408e-b99c-89cd312e9b78" (UID: "4e630456-41c9-408e-b99c-89cd312e9b78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:21.953774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.953744 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e630456-41c9-408e-b99c-89cd312e9b78-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.953774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.953770 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e630456-41c9-408e-b99c-89cd312e9b78-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.953774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.953780 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfbr\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-kube-api-access-fqfbr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:21.954045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:21.953789 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e630456-41c9-408e-b99c-89cd312e9b78-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:22.150298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.150263 5938 generic.go:356] "Generic (PLEG): container finished" podID="4e630456-41c9-408e-b99c-89cd312e9b78" containerID="2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318" exitCode=0 Oct 30 01:03:22.150444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.150311 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" Oct 30 01:03:22.150444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.150349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" event={"ID":"4e630456-41c9-408e-b99c-89cd312e9b78","Type":"ContainerDied","Data":"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318"} Oct 30 01:03:22.150444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.150400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8" event={"ID":"4e630456-41c9-408e-b99c-89cd312e9b78","Type":"ContainerDied","Data":"0e2d608ad88af5a5c2697ca16b0cc3191717c68bd3ff92e4b45c3db1318b3a03"} Oct 30 01:03:22.150444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.150424 5938 scope.go:122] "RemoveContainer" containerID="2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318" Oct 30 01:03:22.161699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.161682 5938 scope.go:122] "RemoveContainer" containerID="2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318" Oct 30 01:03:22.162011 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:22.161993 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318\": container with ID starting with 2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318 not found: ID does not exist" containerID="2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318" Oct 30 01:03:22.162064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.162018 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318"} err="failed to get container status \"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318\": rpc error: code = NotFound desc = could not find container \"2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318\": container with ID starting with 2aac57853ec318f147e516e105cb242cc298fc69bd364718d667da4712b8d318 not found: ID does not exist" Oct 30 01:03:22.206056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.206034 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 01:03:22.217347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.217329 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-jdlt8"] Oct 30 01:03:22.277208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:22.277161 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:23.882662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:23.882613 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" path="/var/lib/kubelet/pods/4e630456-41c9-408e-b99c-89cd312e9b78/volumes" Oct 30 01:03:23.925365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:23.925318 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:25.723936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:25.723900 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:26.222575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:26.222540 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 01:03:26.272932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:26.272713 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 01:03:26.402357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:26.402325 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:26.544599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:26.544516 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:27.418375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:27.418340 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:27.559384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:27.559358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:27.703694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:27.703610 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:28.839198 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:28.839162 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:03:29.385340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:29.385301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:30.903304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:30.903270 5938 scope.go:122] "RemoveContainer" containerID="ca03814a44410ef772c7df306ed491b2804a5f0e7496681dfe02336224066bda" Oct 30 01:03:31.146483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:31.146450 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:31.223215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:31.223133 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" containerID="cri-o://a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf" gracePeriod=25 Oct 30 01:03:31.273380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:31.273345 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" containerID="cri-o://ae3367542c00678e6fee8ae8319b0d822712c3535e32e21c3497a4d91460664f" gracePeriod=25 Oct 30 01:03:32.834260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:32.834230 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:34.648797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:34.648760 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:34.853474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:34.853437 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:03:34.853639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:34.853549 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:03:36.507899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:36.507825 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:36.686882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:36.684564 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:36.829530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:36.829443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:38.159930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:38.153215 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 01:03:38.194885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:38.180850 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:03:38.529406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:38.529322 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:38.947849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:38.947809 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.144347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.144308 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.318961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.318855 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.465607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.465568 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.610467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.610432 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.807771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.807731 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.970181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:39.970092 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:41.317440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:41.317408 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:43.173044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:43.173010 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:44.853525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:44.853485 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:03:44.854052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:44.853529 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:03:45.065262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:45.065220 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:46.354436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.354401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:46.506121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.506085 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:46.686528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.686446 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 01:03:46.686815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.686760 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" containerID="cri-o://5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4" gracePeriod=30 Oct 30 01:03:46.747143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.747111 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:46.898885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:46.889312 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:48.625535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.625503 5938 generic.go:356] "Generic (PLEG): container finished" podID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerID="a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5" exitCode=0 Oct 30 01:03:48.625975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.625539 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerDied","Data":"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5"} Oct 30 01:03:48.625975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.625933 5938 scope.go:122] "RemoveContainer" containerID="a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5" Oct 30 01:03:48.627560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.627529 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:48.627673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.627595 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:48.627673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.627629 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:03:48.832132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:48.832093 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:49.646585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:49.646545 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerStarted","Data":"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe"} Oct 30 01:03:49.990945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:49.984219 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.123075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:50.123045 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.369888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:50.369838 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:03:50.370069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:50.370043 5938 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:03:50.370115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:50.370098 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 01:03:50.904668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:50.898693 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:51.370007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.369976 5938 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:03:51.370137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.370034 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 01:03:51.667839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.667812 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 01:03:51.671241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.671210 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerID="a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf" exitCode=0 Oct 30 01:03:51.671343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.671247 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" Oct 30 01:03:51.671343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.671303 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" event={"ID":"3f0e83c1-6a4a-4a69-81a8-e352193bba3b","Type":"ContainerDied","Data":"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf"} Oct 30 01:03:51.671415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.671352 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9" event={"ID":"3f0e83c1-6a4a-4a69-81a8-e352193bba3b","Type":"ContainerDied","Data":"91cec4bd0c4681f51908707b2830244cf014fcee1b7a6fff25712c34b89d97f5"} Oct 30 01:03:51.671415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.671381 5938 scope.go:122] "RemoveContainer" containerID="a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf" Oct 30 01:03:51.674509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.674356 5938 generic.go:356] "Generic (PLEG): container finished" podID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerID="ae3367542c00678e6fee8ae8319b0d822712c3535e32e21c3497a4d91460664f" exitCode=0 Oct 30 01:03:51.674509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.674432 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" event={"ID":"728bfdfc-545c-4608-b7ca-cd42133ac9cf","Type":"ContainerDied","Data":"ae3367542c00678e6fee8ae8319b0d822712c3535e32e21c3497a4d91460664f"} Oct 30 01:03:51.683736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.683718 5938 scope.go:122] "RemoveContainer" containerID="a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf" Oct 30 01:03:51.709888 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:51.699043 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf\": container with ID starting with a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf not found: ID does not exist" containerID="a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf" Oct 30 01:03:51.709888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.699082 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf"} err="failed to get container status \"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf\": rpc error: code = NotFound desc = could not find container \"a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf\": container with ID starting with a85403e5cc11d9182c313d5e4d740c9b3df95962f5d38e3c18dbb6f04a61ddcf not found: ID does not exist" Oct 30 01:03:51.774072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.773505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\") pod \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " Oct 30 01:03:51.774072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.773559 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\" (UniqueName: \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\") pod \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\" (UID: \"3f0e83c1-6a4a-4a69-81a8-e352193bba3b\") " Oct 30 01:03:51.774072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.773978 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume" pod "3f0e83c1-6a4a-4a69-81a8-e352193bba3b" (UID: "3f0e83c1-6a4a-4a69-81a8-e352193bba3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:51.774346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.774187 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:51.776974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.776939 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d" pod "3f0e83c1-6a4a-4a69-81a8-e352193bba3b" (UID: "3f0e83c1-6a4a-4a69-81a8-e352193bba3b"). InnerVolumeSpecName "kube-api-access-xdx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:51.823102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.823075 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 01:03:51.875690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.875654 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\" (UniqueName: \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\") pod \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " Oct 30 01:03:51.875925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.875748 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\") pod \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\" (UID: \"728bfdfc-545c-4608-b7ca-cd42133ac9cf\") " Oct 30 01:03:51.876548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.876522 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume" pod "728bfdfc-545c-4608-b7ca-cd42133ac9cf" (UID: "728bfdfc-545c-4608-b7ca-cd42133ac9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:51.876695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.876676 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdx7d\" (UniqueName: \"kubernetes.io/projected/3f0e83c1-6a4a-4a69-81a8-e352193bba3b-kube-api-access-xdx7d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:51.876755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.876703 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/728bfdfc-545c-4608-b7ca-cd42133ac9cf-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:51.883920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.881352 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65" pod "728bfdfc-545c-4608-b7ca-cd42133ac9cf" (UID: "728bfdfc-545c-4608-b7ca-cd42133ac9cf"). InnerVolumeSpecName "kube-api-access-rhl65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:51.977931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:51.977893 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhl65\" (UniqueName: \"kubernetes.io/projected/728bfdfc-545c-4608-b7ca-cd42133ac9cf-kube-api-access-rhl65\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.019845 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.019812 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 01:03:52.026826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.026756 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-481ab7-7fc587dfd6-lcrb9"] Oct 30 01:03:52.059069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.059049 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 01:03:52.180688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180653 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.180916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180698 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.180916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180746 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.180916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180766 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.181102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180918 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.181102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180951 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:52.181102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.180960 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.181265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181085 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:52.181265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181150 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:52.181265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181160 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:52.181265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181170 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.181265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.181538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181336 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\") pod \"fa93d7f0-cff8-41be-9296-ac152e0cd370\" (UID: \"fa93d7f0-cff8-41be-9296-ac152e0cd370\") " Oct 30 01:03:52.182007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.181976 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:52.182172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.182091 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.182172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.182114 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa93d7f0-cff8-41be-9296-ac152e0cd370-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.182172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.182128 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.182172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.182142 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.182172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.182156 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.183319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.183292 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:52.183531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.183509 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "kube-api-access-6jh7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:52.183850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.183824 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:03:52.204016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.203967 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo" pod "fa93d7f0-cff8-41be-9296-ac152e0cd370" (UID: "fa93d7f0-cff8-41be-9296-ac152e0cd370"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:03:52.283432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.283369 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.283432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.283394 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jh7f\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-kube-api-access-6jh7f\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.283432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.283406 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.283432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.283427 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa93d7f0-cff8-41be-9296-ac152e0cd370-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:03:52.370181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.370144 5938 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:03:52.370345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.370203 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 01:03:52.710155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.710119 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" event={"ID":"728bfdfc-545c-4608-b7ca-cd42133ac9cf","Type":"ContainerDied","Data":"b6c37bcb5dff958132ba830eb29d5a3da5fe6fe2af627266e5918798c06310af"} Oct 30 01:03:52.710375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.710128 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl" Oct 30 01:03:52.710375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.710118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:52.710375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.710210 5938 scope.go:122] "RemoveContainer" containerID="ae3367542c00678e6fee8ae8319b0d822712c3535e32e21c3497a4d91460664f" Oct 30 01:03:52.712336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.712295 5938 generic.go:356] "Generic (PLEG): container finished" podID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerID="5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4" exitCode=0 Oct 30 01:03:52.712481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.712374 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" Oct 30 01:03:52.712481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.712415 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerDied","Data":"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4"} Oct 30 01:03:52.712618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.712538 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w" event={"ID":"fa93d7f0-cff8-41be-9296-ac152e0cd370","Type":"ContainerDied","Data":"9c47aef421ff2b0060f3df2cdcb4661985908d1ab76a5b27cf0f20e99c9e87b0"} Oct 30 01:03:52.755924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.752505 5938 scope.go:122] "RemoveContainer" containerID="5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4" Oct 30 01:03:52.768228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.768201 5938 scope.go:122] "RemoveContainer" containerID="0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100" Oct 30 01:03:52.781259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.781231 5938 scope.go:122] "RemoveContainer" containerID="5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4" Oct 30 01:03:52.781582 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:52.781560 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4\": container with ID starting with 5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4 not found: ID does not exist" containerID="5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4" Oct 30 01:03:52.781671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.781591 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4"} err="failed to get container status \"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4\": rpc error: code = NotFound desc = could not find container \"5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4\": container with ID starting with 5527f4e54d3783f6859ea1a03217ca28f9069d5e7235bd9e45d762d8d9bcb3e4 not found: ID does not exist" Oct 30 01:03:52.781671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.781612 5938 scope.go:122] "RemoveContainer" containerID="0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100" Oct 30 01:03:52.781938 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:52.781920 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100\": container with ID starting with 0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100 not found: ID does not exist" containerID="0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100" Oct 30 01:03:52.782027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.781945 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100"} err="failed to get container status \"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100\": rpc error: code = NotFound desc = could not find container \"0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100\": container with ID starting with 0d4dfc7e66970bd0c415b536892c5fde367fb102dd66b610d26b9a06e174e100 not found: ID does not exist" Oct 30 01:03:52.814019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.813989 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 01:03:52.817908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.817891 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-481ab7-6c7d8bfdb7-b5zsl"] Oct 30 01:03:52.834588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.834561 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 01:03:52.840049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:52.840029 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-481ab7-openshift-default-59958f667c-kjc7w"] Oct 30 01:03:53.370567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:53.370530 5938 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:03:53.371112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:53.370591 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 01:03:53.878822 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:53.878789 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" path="/var/lib/kubelet/pods/3f0e83c1-6a4a-4a69-81a8-e352193bba3b/volumes" Oct 30 01:03:53.879255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:53.879238 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" path="/var/lib/kubelet/pods/728bfdfc-545c-4608-b7ca-cd42133ac9cf/volumes" Oct 30 01:03:53.879688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:53.879675 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" path="/var/lib/kubelet/pods/fa93d7f0-cff8-41be-9296-ac152e0cd370/volumes" Oct 30 01:03:54.370571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:54.370536 5938 patch_prober.go:29] interesting pod/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:03:54.371101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:54.370606 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 01:03:54.438584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:54.438536 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:55.393893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:55.393408 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:03:55.394351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:55.394023 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:03:55.396113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:55.396088 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:03:56.169269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:56.169212 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.309692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:56.309648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.449155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:56.449069 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:57.449042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:57.449008 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 01:03:57.478751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:57.478723 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 01:03:57.633950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:57.633918 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:57.781581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:57.781498 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:58.297699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:03:58.297666 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:58.836247 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:03:58.836214 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:04:00.182821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:00.182768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:00.652529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:00.652499 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:01.908616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:01.908571 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:02.449448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.449413 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" podUID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" containerName="payload-processing" containerID="cri-o://b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e" gracePeriod=25 Oct 30 01:04:02.479291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.479264 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" containerID="cri-o://21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186" gracePeriod=25 Oct 30 01:04:02.734481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.734445 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 01:04:02.779025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.778973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\") pod \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " Oct 30 01:04:02.779189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.779108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\" (UniqueName: \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\") pod \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\" (UID: \"fb9ad999-ac50-491f-9af1-a51a76ed08c2\") " Oct 30 01:04:02.779403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.779374 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume" pod "fb9ad999-ac50-491f-9af1-a51a76ed08c2" (UID: "fb9ad999-ac50-491f-9af1-a51a76ed08c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:02.779820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.779799 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb9ad999-ac50-491f-9af1-a51a76ed08c2-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:02.781439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.781415 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n" pod "fb9ad999-ac50-491f-9af1-a51a76ed08c2" (UID: "fb9ad999-ac50-491f-9af1-a51a76ed08c2"). InnerVolumeSpecName "kube-api-access-lr59n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:02.799790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.799762 5938 generic.go:356] "Generic (PLEG): container finished" podID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" containerID="b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e" exitCode=0 Oct 30 01:04:02.799950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.799815 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" Oct 30 01:04:02.799950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.799853 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" event={"ID":"fb9ad999-ac50-491f-9af1-a51a76ed08c2","Type":"ContainerDied","Data":"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e"} Oct 30 01:04:02.799950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.799924 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k" event={"ID":"fb9ad999-ac50-491f-9af1-a51a76ed08c2","Type":"ContainerDied","Data":"d1241f41cb63e489baab5448d5e91ab1d60c71af68b7a5b641cedd6f277c140f"} Oct 30 01:04:02.799950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.799939 5938 scope.go:122] "RemoveContainer" containerID="b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e" Oct 30 01:04:02.813728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.813706 5938 scope.go:122] "RemoveContainer" containerID="b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e" Oct 30 01:04:02.814032 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:02.814013 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e\": container with ID starting with b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e not found: ID does not exist" containerID="b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e" Oct 30 01:04:02.814123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.814037 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e"} err="failed to get container status \"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e\": rpc error: code = NotFound desc = could not find container \"b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e\": container with ID starting with b2ec10247a4feefd5fb862fa5d8c360f4c95a7cb4f1c949805a4c153c26b9b7e not found: ID does not exist" Oct 30 01:04:02.851563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.851533 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 01:04:02.856236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.856214 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-03d892-696b99c46-mb79k"] Oct 30 01:04:02.880972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:02.880942 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr59n\" (UniqueName: \"kubernetes.io/projected/fb9ad999-ac50-491f-9af1-a51a76ed08c2-kube-api-access-lr59n\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:03.696546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:03.696505 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.878605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:03.878575 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" path="/var/lib/kubelet/pods/fb9ad999-ac50-491f-9af1-a51a76ed08c2/volumes" Oct 30 01:04:05.282048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:05.281975 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:06.921295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:06.921246 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:08.575078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:08.575045 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:08.715938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:08.715846 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:09.242828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:09.242747 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:04:10.346156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:10.346116 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:12.085604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:12.085518 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:13.732922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:13.732854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.958085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:14.958052 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:15.507746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:15.507665 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:17.389698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:17.389637 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:19.032735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:19.032648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:19.242476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:19.242440 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:04:20.765146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:20.765079 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:22.548798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.548751 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:22.777708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.777682 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 01:04:22.795542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.795519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\") pod \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " Oct 30 01:04:22.795682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.795668 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\" (UniqueName: \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\") pod \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\" (UID: \"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633\") " Oct 30 01:04:22.795921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.795899 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume" pod "ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" (UID: "ba6b9f4a-72b6-45ed-8d08-f8ed9244a633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:22.796299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.796283 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:22.797794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.797771 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654" pod "ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" (UID: "ba6b9f4a-72b6-45ed-8d08-f8ed9244a633"). InnerVolumeSpecName "kube-api-access-mz654". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:22.898312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.898284 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz654\" (UniqueName: \"kubernetes.io/projected/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633-kube-api-access-mz654\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:22.967409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.967379 5938 generic.go:356] "Generic (PLEG): container finished" podID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerID="21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186" exitCode=0 Oct 30 01:04:22.967557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.967424 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" Oct 30 01:04:22.967557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.967472 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" event={"ID":"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633","Type":"ContainerDied","Data":"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186"} Oct 30 01:04:22.967557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.967523 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f" event={"ID":"ba6b9f4a-72b6-45ed-8d08-f8ed9244a633","Type":"ContainerDied","Data":"fa613ac6979d8e8ea4db9f8aedec60db24fd3e505603d33f953bcb3de96cbf2c"} Oct 30 01:04:22.967557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.967545 5938 scope.go:122] "RemoveContainer" containerID="21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186" Oct 30 01:04:22.982799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.982782 5938 scope.go:122] "RemoveContainer" containerID="21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186" Oct 30 01:04:22.983211 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:22.983188 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186\": container with ID starting with 21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186 not found: ID does not exist" containerID="21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186" Oct 30 01:04:22.983296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:22.983218 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186"} err="failed to get container status \"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186\": rpc error: code = NotFound desc = could not find container \"21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186\": container with ID starting with 21243e172684c6e1138a42f64f0a11e4eb7085d511a78565e56817d4f445b186 not found: ID does not exist" Oct 30 01:04:23.021540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:23.021483 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 01:04:23.025095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:23.025073 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-03d892-8674cc764c-gtl5f"] Oct 30 01:04:23.878988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:23.878953 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" path="/var/lib/kubelet/pods/ba6b9f4a-72b6-45ed-8d08-f8ed9244a633/volumes" Oct 30 01:04:24.433797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:24.433750 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:25.923373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:25.923326 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:26.071858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:26.071827 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:26.479566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:26.479525 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:26.621041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:26.621003 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:26.773210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:26.773136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.219444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:27.219403 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.359945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:27.359895 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.706195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:27.704441 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:28.787519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:28.787489 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:28.860039 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:28.860008 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:04:30.568904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:30.568854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:31.006725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:31.006700 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:31.769602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:31.769422 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:31.923888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:31.923445 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:33.076823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:33.076788 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:33.471109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:33.471077 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:33.620912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:33.620876 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.234376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.234341 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.382674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.382625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.549365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.549288 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 01:04:34.549655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.549608 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" containerID="cri-o://bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0" gracePeriod=30 Oct 30 01:04:34.611147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.611117 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.752486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:34.752448 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.400123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:35.400083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.692425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:35.692339 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.842734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:35.842683 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:36.005419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:36.005331 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:36.424550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:36.424519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:37.573492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:37.573461 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:04:37.573919 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:37.573520 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:04:37.798239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:37.798197 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:38.133848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:38.133812 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 01:04:38.134123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:38.134094 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" containerID="cri-o://83abc6044d62509766cf0f30947d3101948718ae9c1e8190ff00ce99d8d06baa" gracePeriod=30 Oct 30 01:04:39.092810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.092735 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:39.226799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.226767 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:39.554381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.554350 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:39.842727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.842703 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 01:04:39.953291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953258 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953321 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953357 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953388 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953478 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953507 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953550 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:39.953749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953596 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.953994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.953840 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\") pod \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\" (UID: \"1b943865-a7ab-462c-b1ce-3f85e0146fdf\") " Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954316 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954521 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954719 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954742 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954757 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:39.954816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.954770 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:39.955447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.955411 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:39.957479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.957076 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "kube-api-access-khgl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:39.957479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.957280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:39.957479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.957416 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:39.958384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:39.958358 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy" pod "1b943865-a7ab-462c-b1ce-3f85e0146fdf" (UID: "1b943865-a7ab-462c-b1ce-3f85e0146fdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:40.056442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.056419 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.056442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.056440 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.056571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.056451 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgl9\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-kube-api-access-khgl9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.056571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.056460 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.056571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.056468 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b943865-a7ab-462c-b1ce-3f85e0146fdf-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:40.386902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.386849 5938 generic.go:356] "Generic (PLEG): container finished" podID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerID="bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0" exitCode=0 Oct 30 01:04:40.387397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.387058 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" Oct 30 01:04:40.388183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.387062 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerDied","Data":"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0"} Oct 30 01:04:40.388410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.388222 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6" event={"ID":"1b943865-a7ab-462c-b1ce-3f85e0146fdf","Type":"ContainerDied","Data":"8d363718cfc6a5ea1d9b96126ec9622f9f881cca096ecaabb91cf95088e87d11"} Oct 30 01:04:40.388410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.388261 5938 scope.go:122] "RemoveContainer" containerID="bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0" Oct 30 01:04:40.402837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.402794 5938 scope.go:122] "RemoveContainer" containerID="85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596" Oct 30 01:04:40.418280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.418261 5938 scope.go:122] "RemoveContainer" containerID="bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0" Oct 30 01:04:40.418560 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:40.418542 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0\": container with ID starting with bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0 not found: ID does not exist" containerID="bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0" Oct 30 01:04:40.418642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.418566 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0"} err="failed to get container status \"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0\": rpc error: code = NotFound desc = could not find container \"bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0\": container with ID starting with bf059ae2505f5ba5d0cb62b11014f2bd02c92abaf6e344da006f22fef7d07eb0 not found: ID does not exist" Oct 30 01:04:40.418642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.418582 5938 scope.go:122] "RemoveContainer" containerID="85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596" Oct 30 01:04:40.418961 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:40.418908 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596\": container with ID starting with 85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596 not found: ID does not exist" containerID="85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596" Oct 30 01:04:40.418961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.418940 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596"} err="failed to get container status \"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596\": rpc error: code = NotFound desc = could not find container \"85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596\": container with ID starting with 85aff8893c6f24ed98754e5c28a247dc72c75dff05e79c0a6d9c701412b68596 not found: ID does not exist" Oct 30 01:04:40.440209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.440185 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 01:04:40.451636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:40.451614 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8dbe73-openshift-default-6c44d95ddf-cdcq6"] Oct 30 01:04:41.252490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:41.252447 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:41.878856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:41.878815 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" path="/var/lib/kubelet/pods/1b943865-a7ab-462c-b1ce-3f85e0146fdf/volumes" Oct 30 01:04:42.536374 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:42.536313 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:42.686705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:42.686675 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:42.825694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:42.825612 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 01:04:42.825931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:42.825853 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" containerID="cri-o://c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd" gracePeriod=30 Oct 30 01:04:42.889452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:42.889422 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.036898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.034430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.414642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.414614 5938 generic.go:356] "Generic (PLEG): container finished" podID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerID="83abc6044d62509766cf0f30947d3101948718ae9c1e8190ff00ce99d8d06baa" exitCode=0 Oct 30 01:04:43.414792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.414689 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" event={"ID":"539ef6c2-f9e3-4032-ab01-f5d335399e64","Type":"ContainerDied","Data":"83abc6044d62509766cf0f30947d3101948718ae9c1e8190ff00ce99d8d06baa"} Oct 30 01:04:43.414792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.414752 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" event={"ID":"539ef6c2-f9e3-4032-ab01-f5d335399e64","Type":"ContainerDied","Data":"94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f"} Oct 30 01:04:43.414792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.414788 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a82988d2dd0534d4170d295ac555f81e66c9b8646ba86442f264fae0dc808f" Oct 30 01:04:43.427972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.427951 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 01:04:43.524275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524229 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524361 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524402 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524455 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524474 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524501 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524646 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.524827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.524677 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\") pod \"539ef6c2-f9e3-4032-ab01-f5d335399e64\" (UID: \"539ef6c2-f9e3-4032-ab01-f5d335399e64\") " Oct 30 01:04:43.525097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525057 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:43.525531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525401 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.525531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:43.525531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525472 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:43.525531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525478 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:43.526141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.525637 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:43.527654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.527623 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "kube-api-access-2br76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:43.528456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.528432 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:43.528673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.528648 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:43.529252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.529229 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy" pod "539ef6c2-f9e3-4032-ab01-f5d335399e64" (UID: "539ef6c2-f9e3-4032-ab01-f5d335399e64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:43.627002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.626971 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/539ef6c2-f9e3-4032-ab01-f5d335399e64-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.626997 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2br76\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-kube-api-access-2br76\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627007 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627015 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627023 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627031 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627040 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/539ef6c2-f9e3-4032-ab01-f5d335399e64-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.627243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.627047 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/539ef6c2-f9e3-4032-ab01-f5d335399e64-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:43.755104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:43.755026 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.165641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.165611 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:04:44.166607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166581 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" Oct 30 01:04:44.166607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166604 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166622 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166629 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166648 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166656 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166670 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166679 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166703 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166712 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166730 5938 cpu_manager.go:414] "containerMap: removing container" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166737 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166790 5938 cpu_manager.go:414] "containerMap: removing container" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" Oct 30 01:04:44.166798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166800 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166828 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" containerName="payload-processing" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.166836 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" containerName="payload-processing" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167029 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167043 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9ad999-ac50-491f-9af1-a51a76ed08c2" containerName="payload-processing" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167053 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167061 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e630456-41c9-408e-b99c-89cd312e9b78" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167069 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167074 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167081 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6b9f4a-72b6-45ed-8d08-f8ed9244a633" containerName="payload-pre-processing" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167088 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0e83c1-6a4a-4a69-81a8-e352193bba3b" containerName="payload-processing" Oct 30 01:04:44.167553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.167094 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="728bfdfc-545c-4608-b7ca-cd42133ac9cf" containerName="payload-pre-processing" Oct 30 01:04:44.171179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.171157 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.194389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.194359 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:04:44.200417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.200395 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-009e78-openshift-default-dockercfg-c68np\"" Oct 30 01:04:44.234374 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234348 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234408 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234602 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234636 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234732 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234787 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6rh\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.234995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.234976 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.235053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.235014 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.235053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.235042 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.249036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.249010 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.405375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.405337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424298 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424361 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424424 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424465 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6rh\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424630 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424672 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424707 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.424815 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.426857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.425311 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.438994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.436217 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.454888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.441172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.457616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.447784 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.480171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.448348 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.480383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.449200 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.480497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.452398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.531886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.520055 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq" Oct 30 01:04:44.531886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.520454 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.531886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.521469 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6rh\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.582014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.581980 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 01:04:44.586658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.586627 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-tkdxq"] Oct 30 01:04:44.793618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.793540 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:44.913797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:44.888769 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:45.121893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.121646 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:04:45.124753 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:04:45.124724 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323a82af_703b_4887_8832_8187a65a33b2.slice/crio-bddcc732059f9c3ffbbad446105b605881099c9727c092d6d8bb855832f47e93 WatchSource:0}: Error finding container bddcc732059f9c3ffbbad446105b605881099c9727c092d6d8bb855832f47e93: Status 404 returned error can't find the container with id bddcc732059f9c3ffbbad446105b605881099c9727c092d6d8bb855832f47e93 Oct 30 01:04:45.128074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.128036 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:45.128201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.128106 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:45.128201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.128137 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:45.528688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.528607 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerStarted","Data":"2e9f31befbfca22182200433ac9489645db4fa75faf10be1ba5000350aa50799"} Oct 30 01:04:45.528688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.528672 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerStarted","Data":"bddcc732059f9c3ffbbad446105b605881099c9727c092d6d8bb855832f47e93"} Oct 30 01:04:45.573779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.573727 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podStartSLOduration=1.5737138160000002 podStartE2EDuration="1.573713816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:45.57168906 +0000 UTC m=+1937.811924174" watchObservedRunningTime="2026-10-30 01:04:45.573713816 +0000 UTC m=+1937.813948929" Oct 30 01:04:45.647929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.647898 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 01:04:45.679659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.679626 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 01:04:45.794934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.794401 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:45.825422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.825394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:45.906648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:45.894121 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539ef6c2-f9e3-4032-ab01-f5d335399e64" path="/var/lib/kubelet/pods/539ef6c2-f9e3-4032-ab01-f5d335399e64/volumes" Oct 30 01:04:46.026110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.026080 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:46.207037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.206995 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:04:46.208396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.208366 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:46.208396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.208395 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:46.208574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.208415 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:46.208574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.208420 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b943865-a7ab-462c-b1ce-3f85e0146fdf" containerName="istio-proxy" Oct 30 01:04:46.208574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.208540 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa93d7f0-cff8-41be-9296-ac152e0cd370" containerName="istio-proxy" Oct 30 01:04:46.212187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.212169 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.237097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.237071 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ef1084-openshift-default-dockercfg-57vpk\"" Oct 30 01:04:46.240291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.240267 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:04:46.258908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.258582 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.258908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.258656 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.258908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.258880 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.258917 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h48d\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.259067 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.259126 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.259162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.259192 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.259484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.259293 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.382896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382630 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.382896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382703 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.382896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382739 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.382896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382771 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.382896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382890 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.383372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.382966 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.383372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.383018 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.383372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.383236 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.383372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.383273 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h48d\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.434192 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.434916 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.435193 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.447131 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.447382 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.449909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.447622 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.473928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.471836 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.484385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.484355 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.485693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.485667 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h48d\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.531606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.531388 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:46.802068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.801993 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:46.821857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.821776 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:46.895134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.892676 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:46.895134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.892762 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:46.895134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.892801 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:04:46.933760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:46.933725 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:04:47.167020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.159783 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.553450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.553363 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" event={"ID":"a9bcaf69-019c-426c-ac5a-a64f44ac85d5","Type":"ContainerStarted","Data":"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220"} Oct 30 01:04:47.553916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.553475 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:47.554019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.553985 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" event={"ID":"a9bcaf69-019c-426c-ac5a-a64f44ac85d5","Type":"ContainerStarted","Data":"6ecb8a0d84bf27abb22635aa74f8125e938e6fb974edbffc0834773a2fd4110c"} Oct 30 01:04:47.554560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.554533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:04:47.605032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:47.604982 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podStartSLOduration=1.604966662 podStartE2EDuration="1.604966662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:47.600625512 +0000 UTC m=+1939.840860652" watchObservedRunningTime="2026-10-30 01:04:47.604966662 +0000 UTC m=+1939.845201779" Oct 30 01:04:48.157512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.157489 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 01:04:48.228746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.228703 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.228946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.228853 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.228946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.228911 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.228981 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.229039 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.229100 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.229165 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.229345 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.229432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.229417 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\") pod \"d7b0d383-83ea-444e-903f-d50785175255\" (UID: \"d7b0d383-83ea-444e-903f-d50785175255\") " Oct 30 01:04:48.230493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.230460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:48.231011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.230983 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.231260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.231232 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:48.231680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.231295 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:04:48.231680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.231480 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:48.231680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.231565 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:48.233569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.233534 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:04:48.233724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.233706 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:04:48.239052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.239017 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:48.242081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.242051 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8" pod "d7b0d383-83ea-444e-903f-d50785175255" (UID: "d7b0d383-83ea-444e-903f-d50785175255"). InnerVolumeSpecName "kube-api-access-szkh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:04:48.334037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334005 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.334037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334033 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.334037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334044 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.334037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334053 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0d383-83ea-444e-903f-d50785175255-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.335877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334061 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.335877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334069 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b0d383-83ea-444e-903f-d50785175255-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.335877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334078 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkh8\" (UniqueName: \"kubernetes.io/projected/d7b0d383-83ea-444e-903f-d50785175255-kube-api-access-szkh8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.335877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.334086 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b0d383-83ea-444e-903f-d50785175255-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:04:48.376144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.376055 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:48.532314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.532280 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:48.534418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.534392 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:48.534542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.534442 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:48.539493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.539467 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:48.560677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.560650 5938 generic.go:356] "Generic (PLEG): container finished" podID="d7b0d383-83ea-444e-903f-d50785175255" containerID="c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd" exitCode=0 Oct 30 01:04:48.561145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.560711 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" Oct 30 01:04:48.561145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.560731 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerDied","Data":"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd"} Oct 30 01:04:48.561145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.560799 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw" event={"ID":"d7b0d383-83ea-444e-903f-d50785175255","Type":"ContainerDied","Data":"348747f54861c1dacf41f7689577e52bbef0f8c326730080650a39e32a54ad76"} Oct 30 01:04:48.561145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.560836 5938 scope.go:122] "RemoveContainer" containerID="c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd" Oct 30 01:04:48.574024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.574010 5938 scope.go:122] "RemoveContainer" containerID="a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.583993 5938 scope.go:122] "RemoveContainer" containerID="c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:48.584274 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd\": container with ID starting with c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd not found: ID does not exist" containerID="c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.584304 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd"} err="failed to get container status \"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd\": rpc error: code = NotFound desc = could not find container \"c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd\": container with ID starting with c8c68bfd4a305d68b5849bc15f5b52b8b679f38a2bb97aca92793b5be823b5fd not found: ID does not exist" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.584326 5938 scope.go:122] "RemoveContainer" containerID="a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:48.584569 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db\": container with ID starting with a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db not found: ID does not exist" containerID="a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db" Oct 30 01:04:48.608891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.584598 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db"} err="failed to get container status \"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db\": rpc error: code = NotFound desc = could not find container \"a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db\": container with ID starting with a572e46fcc284ca3c27c069dd6524c47f4e78ea31ce3a5261c6bd4bf3bf624db not found: ID does not exist" Oct 30 01:04:48.619369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.619338 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 01:04:48.660959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:48.660886 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-03d892-openshift-default-6d4cdfbfd9-f5kkw"] Oct 30 01:04:49.300977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:49.292328 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:49.532114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:49.532084 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:49.532300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:49.532136 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:49.878916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:49.878885 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b0d383-83ea-444e-903f-d50785175255" path="/var/lib/kubelet/pods/d7b0d383-83ea-444e-903f-d50785175255/volumes" Oct 30 01:04:50.115115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.114803 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:50.261316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.261235 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:50.541363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.541245 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:50.541363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.541306 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:50.650917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.650091 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" containerID="cri-o://ee0fe5931b49605b8dd413b4cf13e68f97d4aa6a964c65aae2ebca7880fdbb77" gracePeriod=25 Oct 30 01:04:50.680570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:50.680532 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" containerID="cri-o://d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf" gracePeriod=25 Oct 30 01:04:51.532642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:51.532608 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:51.533083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:51.532656 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:51.638783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:51.638743 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:51.799941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:51.793607 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:52.532618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:52.532581 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:52.532931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:52.532898 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:53.532404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:53.532370 5938 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:04:53.532718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:53.532426 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 01:04:54.119069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:54.119035 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:54.538245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:54.538166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:54.538692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:54.538666 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:54.539368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:54.539350 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:04:56.141046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:56.141003 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:57.810737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:57.810647 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:58.844484 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:04:58.844444 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:04:59.242367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:59.242330 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:04:59.242540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:59.242399 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:04:59.482856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:04:59.482818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:01.423734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:01.423696 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:02.388431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:02.388397 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 01:05:02.420892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:02.420849 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 01:05:02.575935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:02.575903 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:02.717762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:02.717679 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:03.302201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:03.302169 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:05.255747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:05.255714 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:05.748057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:05.748023 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:07.388825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:07.388781 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" containerID="cri-o://be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b" gracePeriod=25 Oct 30 01:05:07.421972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:07.421924 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" containerID="cri-o://563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418" gracePeriod=25 Oct 30 01:05:07.423819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:07.423790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:09.127121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:09.127088 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:09.242190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:09.242143 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:09.242371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:09.242151 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:09.695464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:09.695430 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:09.695647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:09.695461 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:10.778695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:10.778653 5938 generic.go:356] "Generic (PLEG): container finished" podID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerID="ee0fe5931b49605b8dd413b4cf13e68f97d4aa6a964c65aae2ebca7880fdbb77" exitCode=0 Oct 30 01:05:10.779178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:10.778730 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" event={"ID":"c24cbb2f-fe92-429f-b933-53ab35ee1c9d","Type":"ContainerDied","Data":"ee0fe5931b49605b8dd413b4cf13e68f97d4aa6a964c65aae2ebca7880fdbb77"} Oct 30 01:05:10.781784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:10.781763 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:11.006804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.006777 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 01:05:11.063963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.063936 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 01:05:11.085758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.085727 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\") pod \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " Oct 30 01:05:11.085992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.085972 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\" (UniqueName: \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\") pod \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\" (UID: \"c24cbb2f-fe92-429f-b933-53ab35ee1c9d\") " Oct 30 01:05:11.086134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.086102 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume" pod "c24cbb2f-fe92-429f-b933-53ab35ee1c9d" (UID: "c24cbb2f-fe92-429f-b933-53ab35ee1c9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:11.086761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.086734 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:11.088463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.088435 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm" pod "c24cbb2f-fe92-429f-b933-53ab35ee1c9d" (UID: "c24cbb2f-fe92-429f-b933-53ab35ee1c9d"). InnerVolumeSpecName "kube-api-access-964rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:11.187195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.187171 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\" (UniqueName: \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\") pod \"82c08318-dfc9-4519-8e78-dc4cd767a627\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " Oct 30 01:05:11.187361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.187348 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\") pod \"82c08318-dfc9-4519-8e78-dc4cd767a627\" (UID: \"82c08318-dfc9-4519-8e78-dc4cd767a627\") " Oct 30 01:05:11.187713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.187688 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume" pod "82c08318-dfc9-4519-8e78-dc4cd767a627" (UID: "82c08318-dfc9-4519-8e78-dc4cd767a627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:11.187989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.187971 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964rm\" (UniqueName: \"kubernetes.io/projected/c24cbb2f-fe92-429f-b933-53ab35ee1c9d-kube-api-access-964rm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:11.187989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.187988 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c08318-dfc9-4519-8e78-dc4cd767a627-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:11.189315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.189296 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk" pod "82c08318-dfc9-4519-8e78-dc4cd767a627" (UID: "82c08318-dfc9-4519-8e78-dc4cd767a627"). InnerVolumeSpecName "kube-api-access-swbmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:11.289448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.289395 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbmk\" (UniqueName: \"kubernetes.io/projected/82c08318-dfc9-4519-8e78-dc4cd767a627-kube-api-access-swbmk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:11.787819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.787794 5938 generic.go:356] "Generic (PLEG): container finished" podID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerID="d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf" exitCode=0 Oct 30 01:05:11.788259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.787838 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" Oct 30 01:05:11.788259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.787894 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" event={"ID":"82c08318-dfc9-4519-8e78-dc4cd767a627","Type":"ContainerDied","Data":"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf"} Oct 30 01:05:11.788259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.787956 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb" event={"ID":"82c08318-dfc9-4519-8e78-dc4cd767a627","Type":"ContainerDied","Data":"3e261c4082bc04afec12034a4067861d1e9491b54c51faf89e9138b7407ff60d"} Oct 30 01:05:11.788259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.787993 5938 scope.go:122] "RemoveContainer" containerID="d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf" Oct 30 01:05:11.789590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.789565 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" event={"ID":"c24cbb2f-fe92-429f-b933-53ab35ee1c9d","Type":"ContainerDied","Data":"901b489ba32e1b36126b22e4862b20fd349e4c4c1ee1b6b7026f30412ca55ad7"} Oct 30 01:05:11.789717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.789587 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4" Oct 30 01:05:11.805164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.805145 5938 scope.go:122] "RemoveContainer" containerID="d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf" Oct 30 01:05:11.805737 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:11.805713 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf\": container with ID starting with d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf not found: ID does not exist" containerID="d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf" Oct 30 01:05:11.805823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.805742 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf"} err="failed to get container status \"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf\": rpc error: code = NotFound desc = could not find container \"d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf\": container with ID starting with d422ce16a917bd63d37c8919b02d581f11e8b157518247eacf86908980513dcf not found: ID does not exist" Oct 30 01:05:11.805823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.805760 5938 scope.go:122] "RemoveContainer" containerID="ee0fe5931b49605b8dd413b4cf13e68f97d4aa6a964c65aae2ebca7880fdbb77" Oct 30 01:05:11.840050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.840024 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 01:05:11.850910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.850888 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-84b89f-fd65db4cb-4rpk4"] Oct 30 01:05:11.879948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.879925 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" path="/var/lib/kubelet/pods/c24cbb2f-fe92-429f-b933-53ab35ee1c9d/volumes" Oct 30 01:05:11.880258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.880244 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 01:05:11.880613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:11.880596 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-84b89f-c9c9df96c-7rfbb"] Oct 30 01:05:12.596986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:12.596955 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:13.877941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:13.877910 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" path="/var/lib/kubelet/pods/82c08318-dfc9-4519-8e78-dc4cd767a627/volumes" Oct 30 01:05:14.155378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.155310 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:05:14.156040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156024 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:05:14.156040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156039 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:05:14.156167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156092 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" Oct 30 01:05:14.156167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156098 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" Oct 30 01:05:14.156167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156106 5938 cpu_manager.go:414] "containerMap: removing container" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" Oct 30 01:05:14.156167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156111 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" Oct 30 01:05:14.156301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156205 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:05:14.156301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156223 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:05:14.156301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156228 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24cbb2f-fe92-429f-b933-53ab35ee1c9d" containerName="payload-processing" Oct 30 01:05:14.156301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.156234 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c08318-dfc9-4519-8e78-dc4cd767a627" containerName="payload-pre-processing" Oct 30 01:05:14.161307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.161288 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.179892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.177625 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-73435a-serving-cert\"" Oct 30 01:05:14.195211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.195183 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:05:14.286972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.286943 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:14.325519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.325492 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.325639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.325557 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.325743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.325719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtzc\" (UniqueName: \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.325850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.325834 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.439932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.439280 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.439932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.439453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtzc\" (UniqueName: \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.439932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.439596 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.439932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.439772 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.444263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.440724 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.444263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.441710 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:14.451774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.451747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.461601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.461572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.468950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.468911 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtzc\" (UniqueName: \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\") pod \"maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.476474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.476449 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:14.780761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.780731 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:05:14.781510 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:05:14.781481 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870b8015_3078_4cf3_8d88_a77448233bbe.slice/crio-f5c5f782e3ef6f0c2eb8e685a87e9c9ef670fc626f5a3e87b6c79916c2b01ee8 WatchSource:0}: Error finding container f5c5f782e3ef6f0c2eb8e685a87e9c9ef670fc626f5a3e87b6c79916c2b01ee8: Status 404 returned error can't find the container with id f5c5f782e3ef6f0c2eb8e685a87e9c9ef670fc626f5a3e87b6c79916c2b01ee8 Oct 30 01:05:14.783205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.783182 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:05:14.825015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:14.824982 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" event={"ID":"870b8015-3078-4cf3-8d88-a77448233bbe","Type":"ContainerStarted","Data":"f5c5f782e3ef6f0c2eb8e685a87e9c9ef670fc626f5a3e87b6c79916c2b01ee8"} Oct 30 01:05:15.833552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:15.833519 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" event={"ID":"870b8015-3078-4cf3-8d88-a77448233bbe","Type":"ContainerStarted","Data":"c1ff6e63cfb145648169ef95565c2e045b16cd752ada39c63b9cedc47e325d17"} Oct 30 01:05:15.833997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:15.833613 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:15.877552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:15.877502 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" podStartSLOduration=1.527857598 podStartE2EDuration="1.877489363s" totalImagesPullingTime="349.631765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:15.874043052 +0000 UTC m=+1968.114278189" watchObservedRunningTime="2026-10-30 01:05:15.877489363 +0000 UTC m=+1968.117724502" Oct 30 01:05:15.918138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:15.918106 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:16.082264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:16.082230 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:16.228174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:16.228139 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:18.004796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:18.004756 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:19.547364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:19.547310 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:19.695823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:19.695774 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:19.696153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:19.695826 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:05:19.962905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:19.962819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.080424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:21.080392 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.226175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:21.226146 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.666800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:21.666768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.846599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:21.846574 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:05:23.170750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:23.170717 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:23.758014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:23.757977 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:23.922007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:23.921963 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:24.058450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:24.058358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:24.513645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:24.513613 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:26.084537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:26.084499 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:27.713719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.713691 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 01:05:27.733960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.733936 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:27.784009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.783984 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 01:05:27.809455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.809428 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\" (UniqueName: \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\") pod \"8fdb22ff-c281-42aa-b35f-e5a272594283\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " Oct 30 01:05:27.809610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.809571 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\") pod \"8fdb22ff-c281-42aa-b35f-e5a272594283\" (UID: \"8fdb22ff-c281-42aa-b35f-e5a272594283\") " Oct 30 01:05:27.810028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.809998 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume" pod "8fdb22ff-c281-42aa-b35f-e5a272594283" (UID: "8fdb22ff-c281-42aa-b35f-e5a272594283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:27.810309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.810294 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdb22ff-c281-42aa-b35f-e5a272594283-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:27.811604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.811577 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6" pod "8fdb22ff-c281-42aa-b35f-e5a272594283" (UID: "8fdb22ff-c281-42aa-b35f-e5a272594283"). InnerVolumeSpecName "kube-api-access-nhmg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:27.911498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.911470 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\") pod \"b04a828a-6b01-48a3-bf19-fed27ae4399a\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " Oct 30 01:05:27.911748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.911702 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\" (UniqueName: \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\") pod \"b04a828a-6b01-48a3-bf19-fed27ae4399a\" (UID: \"b04a828a-6b01-48a3-bf19-fed27ae4399a\") " Oct 30 01:05:27.911926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.911887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume" pod "b04a828a-6b01-48a3-bf19-fed27ae4399a" (UID: "b04a828a-6b01-48a3-bf19-fed27ae4399a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:27.912407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.912364 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhmg6\" (UniqueName: \"kubernetes.io/projected/8fdb22ff-c281-42aa-b35f-e5a272594283-kube-api-access-nhmg6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:27.912407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.912387 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b04a828a-6b01-48a3-bf19-fed27ae4399a-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:27.913702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.913682 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc" pod "b04a828a-6b01-48a3-bf19-fed27ae4399a" (UID: "b04a828a-6b01-48a3-bf19-fed27ae4399a"). InnerVolumeSpecName "kube-api-access-jrfrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:27.961185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.961122 5938 generic.go:356] "Generic (PLEG): container finished" podID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerID="563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418" exitCode=0 Oct 30 01:05:27.961366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.961180 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" Oct 30 01:05:27.961366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.961213 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" event={"ID":"b04a828a-6b01-48a3-bf19-fed27ae4399a","Type":"ContainerDied","Data":"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418"} Oct 30 01:05:27.961366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.961270 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746" event={"ID":"b04a828a-6b01-48a3-bf19-fed27ae4399a","Type":"ContainerDied","Data":"fbccecdb221879e2adb5637d91c05f2bc7991a67e6a7d4767eccb8e5900b6aaf"} Oct 30 01:05:27.961366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.961298 5938 scope.go:122] "RemoveContainer" containerID="563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418" Oct 30 01:05:27.962792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.962771 5938 generic.go:356] "Generic (PLEG): container finished" podID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerID="be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b" exitCode=0 Oct 30 01:05:27.962988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.962817 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" Oct 30 01:05:27.963083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.962981 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" event={"ID":"8fdb22ff-c281-42aa-b35f-e5a272594283","Type":"ContainerDied","Data":"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b"} Oct 30 01:05:27.963083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.963026 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx" event={"ID":"8fdb22ff-c281-42aa-b35f-e5a272594283","Type":"ContainerDied","Data":"b71dbac305bccd8f456e917d2befae3276dcd59fb10359a57838d75718518308"} Oct 30 01:05:27.973990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.973966 5938 scope.go:122] "RemoveContainer" containerID="563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418" Oct 30 01:05:27.977595 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:27.977569 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418\": container with ID starting with 563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418 not found: ID does not exist" containerID="563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418" Oct 30 01:05:27.977688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.977607 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418"} err="failed to get container status \"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418\": rpc error: code = NotFound desc = could not find container \"563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418\": container with ID starting with 563cfb92ad28c32dca858e3b42583bdc806ff05fd12ff10a14b33d56846ee418 not found: ID does not exist" Oct 30 01:05:27.977688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.977634 5938 scope.go:122] "RemoveContainer" containerID="be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b" Oct 30 01:05:27.988470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.988450 5938 scope.go:122] "RemoveContainer" containerID="be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b" Oct 30 01:05:27.988727 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:27.988704 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b\": container with ID starting with be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b not found: ID does not exist" containerID="be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b" Oct 30 01:05:27.988801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:27.988738 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b"} err="failed to get container status \"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b\": rpc error: code = NotFound desc = could not find container \"be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b\": container with ID starting with be15716314d5718fa0d77487dbfdeb5549525fc52f2d1ffd35b1ee79dcc1721b not found: ID does not exist" Oct 30 01:05:28.013914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:28.013887 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfrc\" (UniqueName: \"kubernetes.io/projected/b04a828a-6b01-48a3-bf19-fed27ae4399a-kube-api-access-jrfrc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:28.015570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:28.015552 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 01:05:28.020840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:28.020818 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d2a70-cd9f5c85b-k28lx"] Oct 30 01:05:28.050154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:28.050120 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 01:05:28.053221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:28.053198 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d2a70-7bc7f566f9-rq746"] Oct 30 01:05:28.836936 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:28.836835 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:05:29.596067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:29.596027 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:29.762712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:29.748253 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:29.879363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:29.879326 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" path="/var/lib/kubelet/pods/8fdb22ff-c281-42aa-b35f-e5a272594283/volumes" Oct 30 01:05:29.879935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:29.879918 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" path="/var/lib/kubelet/pods/b04a828a-6b01-48a3-bf19-fed27ae4399a/volumes" Oct 30 01:05:29.890648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:29.890623 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.013525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.013494 5938 scope.go:122] "RemoveContainer" containerID="83abc6044d62509766cf0f30947d3101948718ae9c1e8190ff00ce99d8d06baa" Oct 30 01:05:31.076831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.076805 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.220938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.220905 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.381338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.381307 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:05:31.381632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.381590 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" containerID="cri-o://3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de" gracePeriod=30 Oct 30 01:05:31.440242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.440207 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.700949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:31.700847 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:32.954404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:32.954361 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:33.603350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:33.603316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:33.751306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:33.751277 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:33.900521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:33.900445 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:34.312291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:34.312211 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:35.634733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:35.634699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:36.671623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.671596 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:05:36.776671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.776505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.776671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.776621 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.777487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777027 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.777487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777123 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.777487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.777487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777336 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.777487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777368 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.779035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777838 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.779035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.777908 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\") pod \"56c9dda2-2849-413b-b7ee-04f30f19a310\" (UID: \"56c9dda2-2849-413b-b7ee-04f30f19a310\") " Oct 30 01:05:36.779304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.779278 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:36.779798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.779738 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:36.780132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.780078 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:36.780479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.780455 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:36.780758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.780734 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:36.781732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.781697 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:36.782990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.782964 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "kube-api-access-mwhxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:36.783236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.783210 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:36.784278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.784253 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo" pod "56c9dda2-2849-413b-b7ee-04f30f19a310" (UID: "56c9dda2-2849-413b-b7ee-04f30f19a310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:36.880292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880267 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880295 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880309 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880322 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880336 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880350 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880364 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c9dda2-2849-413b-b7ee-04f30f19a310-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880378 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhxv\" (UniqueName: \"kubernetes.io/projected/56c9dda2-2849-413b-b7ee-04f30f19a310-kube-api-access-mwhxv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:36.880394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:36.880392 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c9dda2-2849-413b-b7ee-04f30f19a310-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:37.035487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.035424 5938 generic.go:356] "Generic (PLEG): container finished" podID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerID="3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de" exitCode=0 Oct 30 01:05:37.035487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.035459 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" Oct 30 01:05:37.035620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.035516 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" event={"ID":"56c9dda2-2849-413b-b7ee-04f30f19a310","Type":"ContainerDied","Data":"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de"} Oct 30 01:05:37.035620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.035558 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws" event={"ID":"56c9dda2-2849-413b-b7ee-04f30f19a310","Type":"ContainerDied","Data":"27b4b4f2640a332e41512a4061a2b4e888fee6211cb15449b3247a4d66eb6c5a"} Oct 30 01:05:37.035620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.035587 5938 scope.go:122] "RemoveContainer" containerID="3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de" Oct 30 01:05:37.049571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.049554 5938 scope.go:122] "RemoveContainer" containerID="3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de" Oct 30 01:05:37.049922 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:37.049889 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de\": container with ID starting with 3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de not found: ID does not exist" containerID="3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de" Oct 30 01:05:37.050020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.049930 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de"} err="failed to get container status \"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de\": rpc error: code = NotFound desc = could not find container \"3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de\": container with ID starting with 3d24ff72fa12fd1e3f2143f9e4f68f1cfacbc6ee93b8ffba9dcd765211dc91de not found: ID does not exist" Oct 30 01:05:37.087684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.087661 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:05:37.098969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.098948 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-84b89f-openshift-default-f7d86d855-jldws"] Oct 30 01:05:37.490986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.490951 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:37.880334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:37.880298 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" path="/var/lib/kubelet/pods/56c9dda2-2849-413b-b7ee-04f30f19a310/volumes" Oct 30 01:05:39.180806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:39.180725 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:39.381067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:39.381032 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:40.806173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:40.806130 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:40.942804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:40.942772 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:41.085202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.085107 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 01:05:41.085437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.085378 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" containerID="cri-o://5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021" gracePeriod=30 Oct 30 01:05:41.096497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.096469 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 01:05:41.096692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.096664 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" containerID="cri-o://04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf" gracePeriod=30 Oct 30 01:05:41.101070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.101048 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:41.256859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:41.256828 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:43.026217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:43.026181 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:43.799954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:43.799918 5938 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:05:43.800238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:43.800210 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:05:45.207998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:45.207839 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:46.395545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.395513 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 01:05:46.436954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.436927 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 01:05:46.480639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480605 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.480792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480651 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.480792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480697 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.480792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480741 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480805 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.480925 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481017 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481052 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481060 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.481149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481090 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\") pod \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\" (UID: \"c16ade73-e08d-4fed-baf2-a2d3f11e97a3\") " Oct 30 01:05:46.481377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481156 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:46.482030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481445 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.482153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481668 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.482153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481949 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.482153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.481887 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.482153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.482099 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.483323 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.483296 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "kube-api-access-gsdfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:46.483475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.483460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:46.483854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.483828 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:46.484068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.484049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy" pod "c16ade73-e08d-4fed-baf2-a2d3f11e97a3" (UID: "c16ade73-e08d-4fed-baf2-a2d3f11e97a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.583178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583189 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583209 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583306 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583362 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583445 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583474 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.583541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583611 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583652 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\") pod \"8f729a8b-c623-4934-a3b7-e50288a049ce\" (UID: \"8f729a8b-c623-4934-a3b7-e50288a049ce\") " Oct 30 01:05:46.583906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.583777 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.584064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584039 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.584155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584097 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.584155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584143 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:46.584484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584460 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584486 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584502 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584516 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584530 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584542 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f729a8b-c623-4934-a3b7-e50288a049ce-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584555 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584567 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584581 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584595 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584608 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.584664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.584622 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdfl\" (UniqueName: \"kubernetes.io/projected/c16ade73-e08d-4fed-baf2-a2d3f11e97a3-kube-api-access-gsdfl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.586016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.585977 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:05:46.586383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.586340 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:46.586592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.586571 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:05:46.586775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.586756 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts" pod "8f729a8b-c623-4934-a3b7-e50288a049ce" (UID: "8f729a8b-c623-4934-a3b7-e50288a049ce"). InnerVolumeSpecName "kube-api-access-rbfts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:46.685686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.685617 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.685686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.685644 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.685686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.685662 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f729a8b-c623-4934-a3b7-e50288a049ce-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.685686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.685676 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfts\" (UniqueName: \"kubernetes.io/projected/8f729a8b-c623-4934-a3b7-e50288a049ce-kube-api-access-rbfts\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:46.923422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:46.923380 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:47.092046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.091728 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:47.150998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.150964 5938 generic.go:356] "Generic (PLEG): container finished" podID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerID="04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf" exitCode=0 Oct 30 01:05:47.154123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.151442 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" Oct 30 01:05:47.154123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.151478 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerDied","Data":"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf"} Oct 30 01:05:47.154123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.151534 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg" event={"ID":"8f729a8b-c623-4934-a3b7-e50288a049ce","Type":"ContainerDied","Data":"5d247b6ceb32b3a02b0c369b792923ec0e5bb1aaaf11ae9f7eb77cc48fd858a5"} Oct 30 01:05:47.154123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.151600 5938 scope.go:122] "RemoveContainer" containerID="04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf" Oct 30 01:05:47.166090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.161687 5938 generic.go:356] "Generic (PLEG): container finished" podID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerID="5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021" exitCode=0 Oct 30 01:05:47.166090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.161913 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" event={"ID":"c16ade73-e08d-4fed-baf2-a2d3f11e97a3","Type":"ContainerDied","Data":"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021"} Oct 30 01:05:47.166090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.161966 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" event={"ID":"c16ade73-e08d-4fed-baf2-a2d3f11e97a3","Type":"ContainerDied","Data":"ddd8b3f1b26fa66e6d3b70a58ddb8cb633d0a802b3d7251587727727633420b3"} Oct 30 01:05:47.166090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.162053 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n" Oct 30 01:05:47.202532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.202508 5938 scope.go:122] "RemoveContainer" containerID="108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005" Oct 30 01:05:47.214603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.214399 5938 scope.go:122] "RemoveContainer" containerID="04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf" Oct 30 01:05:47.214681 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:47.214656 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf\": container with ID starting with 04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf not found: ID does not exist" containerID="04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf" Oct 30 01:05:47.214761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.214684 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf"} err="failed to get container status \"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf\": rpc error: code = NotFound desc = could not find container \"04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf\": container with ID starting with 04f79bbd103e47840f17e968e87bf22daca34acc86d03a83375ccae2e333dcdf not found: ID does not exist" Oct 30 01:05:47.214761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.214751 5938 scope.go:122] "RemoveContainer" containerID="108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005" Oct 30 01:05:47.215026 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:47.215007 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005\": container with ID starting with 108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005 not found: ID does not exist" containerID="108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005" Oct 30 01:05:47.215115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.215030 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005"} err="failed to get container status \"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005\": rpc error: code = NotFound desc = could not find container \"108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005\": container with ID starting with 108b56f329bd5ae5e913b97d14bfca3bf160f9b9af3766cfca8f6618e199d005 not found: ID does not exist" Oct 30 01:05:47.215115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.215051 5938 scope.go:122] "RemoveContainer" containerID="5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021" Oct 30 01:05:47.225342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.225326 5938 scope.go:122] "RemoveContainer" containerID="5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021" Oct 30 01:05:47.225635 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:47.225616 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021\": container with ID starting with 5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021 not found: ID does not exist" containerID="5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021" Oct 30 01:05:47.225688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.225641 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021"} err="failed to get container status \"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021\": rpc error: code = NotFound desc = could not find container \"5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021\": container with ID starting with 5e7f9d65364333e9f2a47f7ccdaadf1a8b5b0517f8aa3203eed1b39e957f1021 not found: ID does not exist" Oct 30 01:05:47.238355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.238329 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 01:05:47.249454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.249433 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-5kctg"] Oct 30 01:05:47.284030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.284000 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 01:05:47.284175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.284052 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-vpb7n"] Oct 30 01:05:47.879848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.879817 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" path="/var/lib/kubelet/pods/8f729a8b-c623-4934-a3b7-e50288a049ce/volumes" Oct 30 01:05:47.880379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:47.880363 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" path="/var/lib/kubelet/pods/c16ade73-e08d-4fed-baf2-a2d3f11e97a3/volumes" Oct 30 01:05:48.990878 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:48.990826 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:50.659792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:50.659739 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:52.056238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:52.056205 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 01:05:52.228066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:52.228030 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:52.360631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:52.360599 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:52.517784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:52.517738 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:53.142161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:53.142114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:54.280952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:54.280916 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.005686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:56.005634 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:57.056888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.056829 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" podUID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" containerName="payload-processing" containerID="cri-o://464046499d8d1d34ec0e8883b506bdbe21a61331e6fface0594ce60b6b2b66f3" gracePeriod=25 Oct 30 01:05:57.254561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.254513 5938 generic.go:356] "Generic (PLEG): container finished" podID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" containerID="464046499d8d1d34ec0e8883b506bdbe21a61331e6fface0594ce60b6b2b66f3" exitCode=0 Oct 30 01:05:57.254777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.254663 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" event={"ID":"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f","Type":"ContainerDied","Data":"464046499d8d1d34ec0e8883b506bdbe21a61331e6fface0594ce60b6b2b66f3"} Oct 30 01:05:57.365060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.365031 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 01:05:57.449092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.449060 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:57.455206 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.455180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\" (UniqueName: \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\") pod \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " Oct 30 01:05:57.455367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.455227 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\") pod \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\" (UID: \"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f\") " Oct 30 01:05:57.455749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.455718 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume" pod "d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" (UID: "d00ea3f2-557c-41d3-9a21-5ffa70b65b0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:05:57.457525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.457496 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9" pod "d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" (UID: "d00ea3f2-557c-41d3-9a21-5ffa70b65b0f"). InnerVolumeSpecName "kube-api-access-668m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:05:57.515394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.515371 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 01:05:57.534673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.534645 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 01:05:57.557710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.557679 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668m9\" (UniqueName: \"kubernetes.io/projected/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-kube-api-access-668m9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.557827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.557714 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:05:57.636092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.636054 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:57.816208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:57.816167 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.263096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.263014 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" Oct 30 01:05:58.263096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.263036 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn" event={"ID":"d00ea3f2-557c-41d3-9a21-5ffa70b65b0f","Type":"ContainerDied","Data":"8006d64156b1356224a360078be0044ea6246774a161533806956be23d26a93e"} Oct 30 01:05:58.263593 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.263110 5938 scope.go:122] "RemoveContainer" containerID="464046499d8d1d34ec0e8883b506bdbe21a61331e6fface0594ce60b6b2b66f3" Oct 30 01:05:58.316024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.315999 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 01:05:58.319522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.319496 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5007e1-6b98658bc4-8x5fn"] Oct 30 01:05:58.414375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.411042 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.758567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:58.758508 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.837761 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:05:58.837728 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:05:59.198027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:59.197997 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:59.338442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:59.338405 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:59.828764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:59.828689 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:59.878045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:05:59.878015 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" path="/var/lib/kubelet/pods/d00ea3f2-557c-41d3-9a21-5ffa70b65b0f/volumes" Oct 30 01:06:00.377887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:00.376460 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:02.144301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.144269 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:02.372410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.372380 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:06:02.373082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373065 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" Oct 30 01:06:02.373082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373081 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373096 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" containerName="payload-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373102 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" containerName="payload-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373116 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373121 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b0d383-83ea-444e-903f-d50785175255" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373150 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373155 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373165 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373169 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373177 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373182 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373199 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373204 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373215 5938 cpu_manager.go:414] "containerMap: removing container" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" Oct 30 01:06:02.373215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373220 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373313 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdb22ff-c281-42aa-b35f-e5a272594283" containerName="payload-processing" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373322 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373328 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04a828a-6b01-48a3-bf19-fed27ae4399a" containerName="payload-pre-processing" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373338 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c9dda2-2849-413b-b7ee-04f30f19a310" containerName="istio-proxy" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373348 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16ade73-e08d-4fed-baf2-a2d3f11e97a3" containerName="istio-proxy" Oct 30 01:06:02.373661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.373354 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00ea3f2-557c-41d3-9a21-5ffa70b65b0f" containerName="payload-processing" Oct 30 01:06:02.378363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.378343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.410714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.410643 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:06:02.422809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.422777 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dbs\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.422952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.422893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.422973 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.422990 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.423017 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.423074 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.423101 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.423189 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.423236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.423210 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.516228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.516193 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" containerID="cri-o://0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7" gracePeriod=25 Oct 30 01:06:02.524222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524198 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524235 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524317 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524339 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dbs\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524466 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524541 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524570 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524611 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524718 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.524927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.524903 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.525068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.525048 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.525147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.525079 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.525348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.525326 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.526625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.526598 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.526721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.526683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.535715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.535690 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" containerID="cri-o://66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba" gracePeriod=25 Oct 30 01:06:02.546600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.546580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.546689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.546640 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dbs\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:02.694299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:02.694226 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:03.022110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.022070 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:06:03.023839 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:06:03.023802 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcf613f_3ca3_45e4_a4d2_2c948509d5ae.slice/crio-b7e01da9677e439345b13940a84eb57e3390b158b683afd633ae2cfe847695f8 WatchSource:0}: Error finding container b7e01da9677e439345b13940a84eb57e3390b158b683afd633ae2cfe847695f8: Status 404 returned error can't find the container with id b7e01da9677e439345b13940a84eb57e3390b158b683afd633ae2cfe847695f8 Oct 30 01:06:03.026738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.026705 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:03.026902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.026784 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:03.026902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.026831 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:03.027208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.027184 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:03.314859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.313728 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" event={"ID":"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae","Type":"ContainerStarted","Data":"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee"} Oct 30 01:06:03.314859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.313795 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" event={"ID":"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae","Type":"ContainerStarted","Data":"b7e01da9677e439345b13940a84eb57e3390b158b683afd633ae2cfe847695f8"} Oct 30 01:06:03.359985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.359938 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podStartSLOduration=1.359924131 podStartE2EDuration="1.359924131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:03.357042313 +0000 UTC m=+2015.597277439" watchObservedRunningTime="2026-10-30 01:06:03.359924131 +0000 UTC m=+2015.600159243" Oct 30 01:06:03.694666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.694637 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:03.849223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:03.849162 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:04.710555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:04.710517 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 01:06:04.711121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:04.710574 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 01:06:05.538581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.538541 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:05.538763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.538580 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:05.615971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.615929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:05.697890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.697219 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 01:06:05.697890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.697279 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 01:06:05.775931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:05.775898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:06.063592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:06.063557 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:06:06.063889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:06.063837 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" containerID="cri-o://4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02" gracePeriod=30 Oct 30 01:06:06.144594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:06.144552 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:06.704887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:06.701697 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 01:06:06.704887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:06.701756 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 01:06:07.606895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:07.595610 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:07.707895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:07.707214 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 01:06:07.707895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:07.707284 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 01:06:08.708915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:08.706882 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 01:06:08.708915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:08.706935 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 01:06:09.521984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:09.517178 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:09.911308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:09.911274 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:09.911815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:09.911631 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:09.912466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:09.912441 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:06:11.049892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.046931 5938 patch_prober.go:29] interesting pod/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:06:11.049892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.047007 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:06:11.226331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.226293 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:11.390277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.390255 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:06:11.411679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.411651 5938 generic.go:356] "Generic (PLEG): container finished" podID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerID="4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02" exitCode=0 Oct 30 01:06:11.411810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.411684 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" Oct 30 01:06:11.411810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.411739 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" event={"ID":"b873cc77-195e-44f0-ac99-317f15ae2ee8","Type":"ContainerDied","Data":"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02"} Oct 30 01:06:11.411810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.411794 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl" event={"ID":"b873cc77-195e-44f0-ac99-317f15ae2ee8","Type":"ContainerDied","Data":"070d693508bf8897a6ffe863aa9dad4bd39dc0ca3b42d20e6336d95ec32dec82"} Oct 30 01:06:11.412003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.411821 5938 scope.go:122] "RemoveContainer" containerID="4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02" Oct 30 01:06:11.424577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.424557 5938 scope.go:122] "RemoveContainer" containerID="4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02" Oct 30 01:06:11.424856 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:11.424822 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02\": container with ID starting with 4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02 not found: ID does not exist" containerID="4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02" Oct 30 01:06:11.424987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.424853 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02"} err="failed to get container status \"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02\": rpc error: code = NotFound desc = could not find container \"4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02\": container with ID starting with 4ad2100e0fe419304509f9a23ec26b7436cf9ba29c1358c9ecd09e1d5aa14b02 not found: ID does not exist" Oct 30 01:06:11.450343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450320 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450356 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450463 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450515 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450557 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450617 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450801 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.450931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.450856 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\") pod \"b873cc77-195e-44f0-ac99-317f15ae2ee8\" (UID: \"b873cc77-195e-44f0-ac99-317f15ae2ee8\") " Oct 30 01:06:11.451051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.451023 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:11.451210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.451038 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:11.451793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.451761 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.451793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.451795 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.452972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.452926 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:11.453099 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.453029 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:11.453237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.453211 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:11.453332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.453254 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:11.453647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.453626 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "kube-api-access-q5wjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:11.453740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.453698 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:11.455197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.455173 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy" pod "b873cc77-195e-44f0-ac99-317f15ae2ee8" (UID: "b873cc77-195e-44f0-ac99-317f15ae2ee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:11.553400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553371 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553401 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553414 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553428 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553439 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553453 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873cc77-195e-44f0-ac99-317f15ae2ee8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.553464 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wjh\" (UniqueName: \"kubernetes.io/projected/b873cc77-195e-44f0-ac99-317f15ae2ee8-kube-api-access-q5wjh\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:11.757413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.757385 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:06:11.768190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.768167 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51b95e-openshift-default-74bf6d5cb5-tnnfl"] Oct 30 01:06:11.886445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:11.886407 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" path="/var/lib/kubelet/pods/b873cc77-195e-44f0-ac99-317f15ae2ee8/volumes" Oct 30 01:06:13.163995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:13.163939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:15.058748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:15.058705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:15.537804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:15.537770 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:15.538020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:15.537815 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:15.602541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:15.602515 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.077717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.077685 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.554202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.554170 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:06:16.555270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.555248 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" Oct 30 01:06:16.555334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.555275 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" Oct 30 01:06:16.555533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.555518 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b873cc77-195e-44f0-ac99-317f15ae2ee8" containerName="istio-proxy" Oct 30 01:06:16.555573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.555541 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f729a8b-c623-4934-a3b7-e50288a049ce" containerName="istio-proxy" Oct 30 01:06:16.561637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.561613 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.598331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.598309 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-7f92f6-openshift-default-dockercfg-n4bqm\"" Oct 30 01:06:16.609590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.609559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.613598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.613574 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:06:16.627892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.627847 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.627981 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628089 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628250 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628274 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628388 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628370 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nl88\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.628446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.628406 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.729731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729708 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.729851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729739 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.729851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729779 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nl88\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.729851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729800 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729955 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.729992 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730030 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730166 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730211 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730286 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730396 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730583 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.730810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.730791 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.732183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.732163 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.732475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.732455 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.763746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.763698 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nl88\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.769247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.769155 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.912896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.910470 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:16.931931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:16.930383 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.177072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.176805 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:17.278420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.278383 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.315189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.315160 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 01:06:17.347175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.347148 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 01:06:17.428205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.428132 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.499307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.499284 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:06:17.500951 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:06:17.500916 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be1d1ce_3951_43dc_80d6_e72c9c86aee8.slice/crio-e931a2bdec7dad447c6761cb7142c1c4dc8def18a37e3e1cb11012d88b237f69 WatchSource:0}: Error finding container e931a2bdec7dad447c6761cb7142c1c4dc8def18a37e3e1cb11012d88b237f69: Status 404 returned error can't find the container with id e931a2bdec7dad447c6761cb7142c1c4dc8def18a37e3e1cb11012d88b237f69 Oct 30 01:06:17.503910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.503837 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:17.504045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.503919 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:17.504045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.503954 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:17.596402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.596356 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.955586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:17.955551 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:18.475877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:18.475829 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerStarted","Data":"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924"} Oct 30 01:06:18.476273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:18.475899 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerStarted","Data":"e931a2bdec7dad447c6761cb7142c1c4dc8def18a37e3e1cb11012d88b237f69"} Oct 30 01:06:18.534811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:18.534757 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podStartSLOduration=2.534743166 podStartE2EDuration="2.534743166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:18.530740986 +0000 UTC m=+2030.770976091" watchObservedRunningTime="2026-10-30 01:06:18.534743166 +0000 UTC m=+2030.774978273" Oct 30 01:06:19.177745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:19.177708 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:19.183979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:19.183953 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:19.268758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:19.268723 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:19.484025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:19.483948 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:19.484855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:19.484834 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:06:21.042570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:21.042542 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:21.183113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:21.183078 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:22.316424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.316389 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" podUID="4d36a46e-35e7-4ef9-9124-be9f76886052" containerName="payload-processing" containerID="cri-o://750d41af1d8cc63e0b87918a0456c65cd74a3e8e4e83e24ef1be50accbe5eff6" gracePeriod=25 Oct 30 01:06:22.347995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.347963 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" containerID="cri-o://056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e" gracePeriod=25 Oct 30 01:06:22.525010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.524146 5938 generic.go:356] "Generic (PLEG): container finished" podID="4d36a46e-35e7-4ef9-9124-be9f76886052" containerID="750d41af1d8cc63e0b87918a0456c65cd74a3e8e4e83e24ef1be50accbe5eff6" exitCode=0 Oct 30 01:06:22.525969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.525855 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" event={"ID":"4d36a46e-35e7-4ef9-9124-be9f76886052","Type":"ContainerDied","Data":"750d41af1d8cc63e0b87918a0456c65cd74a3e8e4e83e24ef1be50accbe5eff6"} Oct 30 01:06:22.682650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.682394 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 01:06:22.699481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.699368 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:22.806413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.806243 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 01:06:22.854945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.854913 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:22.866237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.866204 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\") pod \"4d36a46e-35e7-4ef9-9124-be9f76886052\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " Oct 30 01:06:22.866400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.866257 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\" (UniqueName: \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\") pod \"4d36a46e-35e7-4ef9-9124-be9f76886052\" (UID: \"4d36a46e-35e7-4ef9-9124-be9f76886052\") " Oct 30 01:06:22.866691 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.866663 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume" pod "4d36a46e-35e7-4ef9-9124-be9f76886052" (UID: "4d36a46e-35e7-4ef9-9124-be9f76886052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:22.867062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.867041 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d36a46e-35e7-4ef9-9124-be9f76886052-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:22.869127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.869086 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82" pod "4d36a46e-35e7-4ef9-9124-be9f76886052" (UID: "4d36a46e-35e7-4ef9-9124-be9f76886052"). InnerVolumeSpecName "kube-api-access-gbm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:22.981888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:22.968427 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\" (UniqueName: \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\") pod \"52226975-0a51-4129-a2d9-1db215e76dac\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " Oct 30 01:06:23.007191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.006378 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\") pod \"52226975-0a51-4129-a2d9-1db215e76dac\" (UID: \"52226975-0a51-4129-a2d9-1db215e76dac\") " Oct 30 01:06:23.012095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.007466 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbm82\" (UniqueName: \"kubernetes.io/projected/4d36a46e-35e7-4ef9-9124-be9f76886052-kube-api-access-gbm82\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.043909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.035187 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume" pod "52226975-0a51-4129-a2d9-1db215e76dac" (UID: "52226975-0a51-4129-a2d9-1db215e76dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:23.065398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.060036 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d" pod "52226975-0a51-4129-a2d9-1db215e76dac" (UID: "52226975-0a51-4129-a2d9-1db215e76dac"). InnerVolumeSpecName "kube-api-access-bvg5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:23.105748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.104892 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 01:06:23.114106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.114080 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvg5d\" (UniqueName: \"kubernetes.io/projected/52226975-0a51-4129-a2d9-1db215e76dac-kube-api-access-bvg5d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.114106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.114103 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52226975-0a51-4129-a2d9-1db215e76dac-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.174043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.174013 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:23.227727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.227693 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\") pod \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " Oct 30 01:06:23.228071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.227791 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\" (UniqueName: \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\") pod \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\" (UID: \"adbed9a1-0ed5-4042-8979-691eb3cb6f0b\") " Oct 30 01:06:23.228234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.228195 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume" pod "adbed9a1-0ed5-4042-8979-691eb3cb6f0b" (UID: "adbed9a1-0ed5-4042-8979-691eb3cb6f0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:23.228632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.228604 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.231303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.231277 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb" pod "adbed9a1-0ed5-4042-8979-691eb3cb6f0b" (UID: "adbed9a1-0ed5-4042-8979-691eb3cb6f0b"). InnerVolumeSpecName "kube-api-access-tsplb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:23.329459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.329425 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsplb\" (UniqueName: \"kubernetes.io/projected/adbed9a1-0ed5-4042-8979-691eb3cb6f0b-kube-api-access-tsplb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:23.534935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.534829 5938 generic.go:356] "Generic (PLEG): container finished" podID="52226975-0a51-4129-a2d9-1db215e76dac" containerID="0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7" exitCode=0 Oct 30 01:06:23.534935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.534904 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" event={"ID":"52226975-0a51-4129-a2d9-1db215e76dac","Type":"ContainerDied","Data":"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7"} Oct 30 01:06:23.535113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.534931 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" Oct 30 01:06:23.535113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.535003 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl" event={"ID":"52226975-0a51-4129-a2d9-1db215e76dac","Type":"ContainerDied","Data":"f45ceb1f07faa6905a020ed94e1ba2959472c79f64668267e17011639fc3b9f6"} Oct 30 01:06:23.535113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.535034 5938 scope.go:122] "RemoveContainer" containerID="0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7" Oct 30 01:06:23.536575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.536550 5938 generic.go:356] "Generic (PLEG): container finished" podID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerID="66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba" exitCode=0 Oct 30 01:06:23.536657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.536590 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" Oct 30 01:06:23.536657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.536636 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" event={"ID":"adbed9a1-0ed5-4042-8979-691eb3cb6f0b","Type":"ContainerDied","Data":"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba"} Oct 30 01:06:23.536783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.536695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d" event={"ID":"adbed9a1-0ed5-4042-8979-691eb3cb6f0b","Type":"ContainerDied","Data":"6bfc484692dc25289fd35fa27c2f30c3369cd5f09256dca07628a1ceb5362076"} Oct 30 01:06:23.538554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.538514 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" event={"ID":"4d36a46e-35e7-4ef9-9124-be9f76886052","Type":"ContainerDied","Data":"6e7ed20c816bcb45fb4aaf03475ff3d57446f21c54bc89ef7008aa380135924d"} Oct 30 01:06:23.538554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.538529 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j" Oct 30 01:06:23.551401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.551379 5938 scope.go:122] "RemoveContainer" containerID="0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7" Oct 30 01:06:23.551984 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:23.551963 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7\": container with ID starting with 0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7 not found: ID does not exist" containerID="0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7" Oct 30 01:06:23.552091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.551990 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7"} err="failed to get container status \"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7\": rpc error: code = NotFound desc = could not find container \"0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7\": container with ID starting with 0f1c347a4a938f6a6e00bd31b5f26147d4b0cf883ff2dd8a3bf9d544551be8d7 not found: ID does not exist" Oct 30 01:06:23.552091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.552007 5938 scope.go:122] "RemoveContainer" containerID="66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba" Oct 30 01:06:23.561670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.561652 5938 scope.go:122] "RemoveContainer" containerID="66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba" Oct 30 01:06:23.561968 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:23.561936 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba\": container with ID starting with 66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba not found: ID does not exist" containerID="66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba" Oct 30 01:06:23.562035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.561978 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba"} err="failed to get container status \"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba\": rpc error: code = NotFound desc = could not find container \"66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba\": container with ID starting with 66c5c35e0edbea69cd2175a4e20ed8af358c742a9240137491e34a88f77aefba not found: ID does not exist" Oct 30 01:06:23.562035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.561999 5938 scope.go:122] "RemoveContainer" containerID="750d41af1d8cc63e0b87918a0456c65cd74a3e8e4e83e24ef1be50accbe5eff6" Oct 30 01:06:23.602809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.602786 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 01:06:23.607353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.607333 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51b95e-58df465794-v4nvl"] Oct 30 01:06:23.654482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.654457 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 01:06:23.659333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.659311 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b1f6ed-579f46c958-7cc8j"] Oct 30 01:06:23.690337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.690312 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 01:06:23.694481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.694458 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51b95e-56d4b8b8b7-9fb4d"] Oct 30 01:06:23.878292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.878266 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d36a46e-35e7-4ef9-9124-be9f76886052" path="/var/lib/kubelet/pods/4d36a46e-35e7-4ef9-9124-be9f76886052/volumes" Oct 30 01:06:23.878686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.878672 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52226975-0a51-4129-a2d9-1db215e76dac" path="/var/lib/kubelet/pods/52226975-0a51-4129-a2d9-1db215e76dac/volumes" Oct 30 01:06:23.879109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:23.879094 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" path="/var/lib/kubelet/pods/adbed9a1-0ed5-4042-8979-691eb3cb6f0b/volumes" Oct 30 01:06:24.072117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:24.072088 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.167688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:24.167614 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:24.955738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:24.955707 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:25.105476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:25.105430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:25.307397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:25.306678 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 01:06:25.307397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:25.307317 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" containerID="cri-o://5546e736bc93a03bb769d91cf1fcaae39eab5f6953478c3382ee9da072ea45c5" gracePeriod=30 Oct 30 01:06:25.362761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:25.362732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:26.740594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:26.740561 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:28.595370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:28.595331 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:28.836603 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:28.836564 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:06:30.090772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.090731 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:30.390750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.390722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:30.604744 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.604715 5938 generic.go:356] "Generic (PLEG): container finished" podID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerID="5546e736bc93a03bb769d91cf1fcaae39eab5f6953478c3382ee9da072ea45c5" exitCode=0 Oct 30 01:06:30.604921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.604794 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerDied","Data":"5546e736bc93a03bb769d91cf1fcaae39eab5f6953478c3382ee9da072ea45c5"} Oct 30 01:06:30.604921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.604908 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" event={"ID":"70f03d8c-6c8b-411f-a279-71375d4597c9","Type":"ContainerDied","Data":"8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f"} Oct 30 01:06:30.605061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.604928 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d857f446aadf89f68e3ec000cea11e53803985ec6b37dcc5ecaa6d5d8bf4d4f" Oct 30 01:06:30.605061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.604942 5938 scope.go:122] "RemoveContainer" containerID="9c85deb20f58c7160245b6feebb9ae3b51daf9d611aaabda3a7a7cb82ecd120e" Oct 30 01:06:30.616803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.616787 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:06:30.634435 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634443 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634492 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634524 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634704 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634743 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.634788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634777 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.635000 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634903 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.635059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635028 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\") pod \"70f03d8c-6c8b-411f-a279-71375d4597c9\" (UID: \"70f03d8c-6c8b-411f-a279-71375d4597c9\") " Oct 30 01:06:30.635447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634772 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:30.635517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.634997 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:30.635517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635164 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:30.635517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635328 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:30.635517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635386 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:30.635720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635701 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.635768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635729 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f03d8c-6c8b-411f-a279-71375d4597c9-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.635768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635745 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.635768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635759 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.635898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.635775 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.636706 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.636683 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:30.637106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.637083 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:30.637106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.637095 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "kube-api-access-d7j7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:30.637285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.637195 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy" pod "70f03d8c-6c8b-411f-a279-71375d4597c9" (UID: "70f03d8c-6c8b-411f-a279-71375d4597c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:30.737261 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.737217 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.737261 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.737258 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.737499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.737273 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f03d8c-6c8b-411f-a279-71375d4597c9-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:30.737499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:30.737288 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7j7p\" (UniqueName: \"kubernetes.io/projected/70f03d8c-6c8b-411f-a279-71375d4597c9-kube-api-access-d7j7p\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:31.613595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:31.613568 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc" Oct 30 01:06:31.706484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:31.706458 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 01:06:31.709927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:31.709898 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5007e1-openshift-default-64c644d8cc-xzwxc"] Oct 30 01:06:31.878574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:31.878510 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" path="/var/lib/kubelet/pods/70f03d8c-6c8b-411f-a279-71375d4597c9/volumes" Oct 30 01:06:32.064319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:32.059915 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:33.782706 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:33.782668 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:34.194887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:34.193970 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:34.987656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:34.987612 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:35.755717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:35.751501 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:37.540114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:37.540083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:39.344234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:39.344200 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:39.482950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:39.482899 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:40.756838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:40.756802 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 01:06:40.757740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:40.757680 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerName="maas-api" containerID="cri-o://3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d" gracePeriod=30 Oct 30 01:06:40.957736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:40.957640 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.209886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.209395 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 01:06:41.270799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.270762 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\" (UniqueName: \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\") pod \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " Oct 30 01:06:41.271089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.271067 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\") pod \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " Oct 30 01:06:41.271271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.271254 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\") pod \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " Oct 30 01:06:41.271418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.271404 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") pod \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\" (UID: \"7170a77a-f354-4f2f-8c6d-b16ef36eea08\") " Oct 30 01:06:41.284392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.284361 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca" pod "7170a77a-f354-4f2f-8c6d-b16ef36eea08" (UID: "7170a77a-f354-4f2f-8c6d-b16ef36eea08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:41.314011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.313975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs" pod "7170a77a-f354-4f2f-8c6d-b16ef36eea08" (UID: "7170a77a-f354-4f2f-8c6d-b16ef36eea08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:41.318411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.318348 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls" pod "7170a77a-f354-4f2f-8c6d-b16ef36eea08" (UID: "7170a77a-f354-4f2f-8c6d-b16ef36eea08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:06:41.369644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.336420 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd" pod "7170a77a-f354-4f2f-8c6d-b16ef36eea08" (UID: "7170a77a-f354-4f2f-8c6d-b16ef36eea08"). InnerVolumeSpecName "kube-api-access-rtwbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:41.389730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.389102 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwbd\" (UniqueName: \"kubernetes.io/projected/7170a77a-f354-4f2f-8c6d-b16ef36eea08-kube-api-access-rtwbd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:41.389730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.389141 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:41.389730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.389155 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7170a77a-f354-4f2f-8c6d-b16ef36eea08-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:41.389730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.389168 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7170a77a-f354-4f2f-8c6d-b16ef36eea08-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:41.531404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.531370 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.679513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.679477 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.742760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.742721 5938 generic.go:356] "Generic (PLEG): container finished" podID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerID="3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d" exitCode=0 Oct 30 01:06:41.742760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.742760 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" Oct 30 01:06:41.743019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.742805 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" event={"ID":"7170a77a-f354-4f2f-8c6d-b16ef36eea08","Type":"ContainerDied","Data":"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d"} Oct 30 01:06:41.743019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.742915 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" event={"ID":"7170a77a-f354-4f2f-8c6d-b16ef36eea08","Type":"ContainerDied","Data":"e464f55aeadd47c88953aca3ee448d7f0418da043e5ce4d1f764be8207275e3e"} Oct 30 01:06:41.743019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.742948 5938 scope.go:122] "RemoveContainer" containerID="3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d" Oct 30 01:06:41.756398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.756370 5938 scope.go:122] "RemoveContainer" containerID="3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d" Oct 30 01:06:41.756684 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:41.756665 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d\": container with ID starting with 3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d not found: ID does not exist" containerID="3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d" Oct 30 01:06:41.756745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.756697 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d"} err="failed to get container status \"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d\": rpc error: code = NotFound desc = could not find container \"3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d\": container with ID starting with 3e890160b41fb1549ab83e169d5fb5b7fce042736d565ae5a1c11db7f388684d not found: ID does not exist" Oct 30 01:06:41.809599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.809561 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 01:06:41.813507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.813484 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7"] Oct 30 01:06:41.825139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.825112 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.878068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:41.878041 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" path="/var/lib/kubelet/pods/7170a77a-f354-4f2f-8c6d-b16ef36eea08/volumes" Oct 30 01:06:42.208888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.208484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.636837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.636808 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 01:06:42.725659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.725572 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\" (UniqueName: \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\") pod \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " Oct 30 01:06:42.725825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.725806 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\") pod \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\" (UID: \"38bc6a7b-9e93-4970-a71b-0abc72258c1e\") " Oct 30 01:06:42.726260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.726230 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume" pod "38bc6a7b-9e93-4970-a71b-0abc72258c1e" (UID: "38bc6a7b-9e93-4970-a71b-0abc72258c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:42.726455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.726434 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38bc6a7b-9e93-4970-a71b-0abc72258c1e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:42.728533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.728507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7" pod "38bc6a7b-9e93-4970-a71b-0abc72258c1e" (UID: "38bc6a7b-9e93-4970-a71b-0abc72258c1e"). InnerVolumeSpecName "kube-api-access-86nj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:42.752569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.752539 5938 generic.go:356] "Generic (PLEG): container finished" podID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerID="056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e" exitCode=0 Oct 30 01:06:42.752700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.752598 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" Oct 30 01:06:42.752700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.752638 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" event={"ID":"38bc6a7b-9e93-4970-a71b-0abc72258c1e","Type":"ContainerDied","Data":"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e"} Oct 30 01:06:42.752819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.752710 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb" event={"ID":"38bc6a7b-9e93-4970-a71b-0abc72258c1e","Type":"ContainerDied","Data":"30b3b259c432f35cc78b15ffc65f8a7a99c806186ae873bbcc79b6bd91209535"} Oct 30 01:06:42.752819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.752751 5938 scope.go:122] "RemoveContainer" containerID="056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e" Oct 30 01:06:42.765233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.765216 5938 scope.go:122] "RemoveContainer" containerID="056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e" Oct 30 01:06:42.765559 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:42.765534 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e\": container with ID starting with 056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e not found: ID does not exist" containerID="056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e" Oct 30 01:06:42.765634 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.765566 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e"} err="failed to get container status \"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e\": rpc error: code = NotFound desc = could not find container \"056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e\": container with ID starting with 056a8f16e8ba0ebdf06e3f8491b7f2e88d0d0617e0e7a56377757b6263cb3a4e not found: ID does not exist" Oct 30 01:06:42.805533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.805513 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 01:06:42.811407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.811380 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b1f6ed-6d7477cd6c-cfgvb"] Oct 30 01:06:42.827338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:42.827314 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86nj7\" (UniqueName: \"kubernetes.io/projected/38bc6a7b-9e93-4970-a71b-0abc72258c1e-kube-api-access-86nj7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:43.051176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:43.051086 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:43.193974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:43.193939 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:43.550001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:43.549959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:43.879520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:43.879475 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" path="/var/lib/kubelet/pods/38bc6a7b-9e93-4970-a71b-0abc72258c1e/volumes" Oct 30 01:06:44.016630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:44.016586 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1f6ed-bf9dd5d46-7nnj7" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.210:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:06:45.129026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:45.128981 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:46.789023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.788939 5938 generic.go:356] "Generic (PLEG): container finished" podID="323a82af-703b-4887-8832-8187a65a33b2" containerID="2e9f31befbfca22182200433ac9489645db4fa75faf10be1ba5000350aa50799" exitCode=0 Oct 30 01:06:46.789023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.788992 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerDied","Data":"2e9f31befbfca22182200433ac9489645db4fa75faf10be1ba5000350aa50799"} Oct 30 01:06:46.789609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.789470 5938 scope.go:122] "RemoveContainer" containerID="2e9f31befbfca22182200433ac9489645db4fa75faf10be1ba5000350aa50799" Oct 30 01:06:46.791260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.791225 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:46.791392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.791312 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:46.791392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.791363 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:46.987834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:46.987801 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:47.553282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:47.553251 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:06:47.821535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:47.821447 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerStarted","Data":"60ab0c98576dd9241ed65f576cb14189c20153321198fa61ac849342278482d9"} Oct 30 01:06:48.432679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.432638 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:48.703934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.700328 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:48.743242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.743208 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:06:48.743477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.743452 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" containerID="cri-o://2db03c89ab30a2d3406697b7d3a22034fa52d7d3547219d8396fc7d032f7cf9d" gracePeriod=30 Oct 30 01:06:48.809583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.809505 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:06:48.810885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.809959 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:06:48.810885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.810011 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 01:06:48.858693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:48.858650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.609907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.607565 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.798891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.798196 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:06:49.798891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.798256 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 01:06:49.967560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.967481 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:06:49.968203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968183 5938 cpu_manager.go:414] "containerMap: removing container" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:49.968203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968202 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968227 5938 cpu_manager.go:414] "containerMap: removing container" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968236 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968243 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4d36a46e-35e7-4ef9-9124-be9f76886052" containerName="payload-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968248 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d36a46e-35e7-4ef9-9124-be9f76886052" containerName="payload-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968314 5938 cpu_manager.go:414] "containerMap: removing container" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968324 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" Oct 30 01:06:49.968350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968350 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerName="maas-api" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968359 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerName="maas-api" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968373 5938 cpu_manager.go:414] "containerMap: removing container" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968379 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968530 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968545 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="52226975-0a51-4129-a2d9-1db215e76dac" containerName="payload-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968552 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d36a46e-35e7-4ef9-9124-be9f76886052" containerName="payload-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968561 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968571 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbed9a1-0ed5-4042-8979-691eb3cb6f0b" containerName="payload-pre-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968578 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bc6a7b-9e93-4970-a71b-0abc72258c1e" containerName="payload-pre-processing" Oct 30 01:06:49.968621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.968585 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7170a77a-f354-4f2f-8c6d-b16ef36eea08" containerName="maas-api" Oct 30 01:06:49.971314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:49.971295 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.006276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.004917 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-93079c-openshift-default-dockercfg-9g4x2\"" Oct 30 01:06:50.025254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.019447 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:06:50.044528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044498 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.044709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044627 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.044709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.044897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044750 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.044897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044790 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.044897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044816 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.045108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044909 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xkq\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.045108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.044967 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.045108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.045084 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.065757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.065723 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:50.147609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.147576 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.148252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.148226 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.149611 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.149589 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.149990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.149970 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.150221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.150203 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.150966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.150946 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.151210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.151190 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xkq\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.151681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.151660 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.151962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.151943 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.153488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.150486 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.153644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.149515 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.153759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.150897 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.153970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.151019 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.191804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.153316 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.192052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.191370 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.193100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.193071 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.209434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.209402 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xkq\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.210142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.210091 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.220793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.220666 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:50.310189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.310154 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:50.428648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.428570 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:50.779491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.779467 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:06:50.781113 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:06:50.781085 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1786f89_495b_42f2_a8d3_7d9c4bbec343.slice/crio-34b7dce5b262500caead8c290b00eee132c00ac4cc3dc3bb2933e07c470958db WatchSource:0}: Error finding container 34b7dce5b262500caead8c290b00eee132c00ac4cc3dc3bb2933e07c470958db: Status 404 returned error can't find the container with id 34b7dce5b262500caead8c290b00eee132c00ac4cc3dc3bb2933e07c470958db Oct 30 01:06:50.783799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.783754 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:50.783938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.783831 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:50.783938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.783881 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:06:50.809031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.809006 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:06:50.809158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.809044 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 01:06:50.843018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:50.842989 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerStarted","Data":"34b7dce5b262500caead8c290b00eee132c00ac4cc3dc3bb2933e07c470958db"} Oct 30 01:06:51.799214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:51.799176 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:06:51.799850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:51.799814 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 01:06:51.851874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:51.851829 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerStarted","Data":"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1"} Oct 30 01:06:52.311375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.311345 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:52.311650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.311375 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:52.344677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.343772 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:52.406596 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.406531 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podStartSLOduration=3.406512611 podStartE2EDuration="3.406512611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:51.909947365 +0000 UTC m=+2064.150182479" watchObservedRunningTime="2026-10-30 01:06:52.406512611 +0000 UTC m=+2064.646747727" Oct 30 01:06:52.809250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.809211 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:06:52.809762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.809275 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 01:06:52.858800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.858772 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:52.859600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:52.859581 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:06:53.238044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.238008 5938 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:06:53.238219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.238071 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:06:53.800140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.800110 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:06:53.800595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.800576 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:06:53.801217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.801200 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:06:53.867003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.866981 5938 generic.go:356] "Generic (PLEG): container finished" podID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerID="2db03c89ab30a2d3406697b7d3a22034fa52d7d3547219d8396fc7d032f7cf9d" exitCode=0 Oct 30 01:06:53.867325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:53.867057 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" event={"ID":"01dc49ed-7fe2-450a-9b07-7c1e2ae56567","Type":"ContainerDied","Data":"2db03c89ab30a2d3406697b7d3a22034fa52d7d3547219d8396fc7d032f7cf9d"} Oct 30 01:06:54.062727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.062701 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:06:54.129547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.129501 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.147445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147407 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147504 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147576 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147635 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147671 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147717 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147803 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.147989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.147959 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.148357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.148020 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\") pod \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\" (UID: \"01dc49ed-7fe2-450a-9b07-7c1e2ae56567\") " Oct 30 01:06:54.149432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.149222 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:54.149752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.149636 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:54.150108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.149916 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:54.150506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.150231 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:54.150846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.150824 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:54.153287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.153260 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:06:54.161709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.161666 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "kube-api-access-5swbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:54.167423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.167337 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:54.167995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.167518 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo" pod "01dc49ed-7fe2-450a-9b07-7c1e2ae56567" (UID: "01dc49ed-7fe2-450a-9b07-7c1e2ae56567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:06:54.250116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250087 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swbt\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-kube-api-access-5swbt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250118 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250132 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250146 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250158 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250170 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250180 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250192 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.250302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.250204 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dc49ed-7fe2-450a-9b07-7c1e2ae56567-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:06:54.403258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.403224 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.707819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.707680 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.875974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.875944 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" Oct 30 01:06:54.875974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.875966 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl" event={"ID":"01dc49ed-7fe2-450a-9b07-7c1e2ae56567","Type":"ContainerDied","Data":"0aaec36f3829aaf701edb971c1ee49bd4aeed586bbd561642079befdf64a2ba0"} Oct 30 01:06:54.876521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.876044 5938 scope.go:122] "RemoveContainer" containerID="2db03c89ab30a2d3406697b7d3a22034fa52d7d3547219d8396fc7d032f7cf9d" Oct 30 01:06:54.943347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.943313 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:06:54.953406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:54.953378 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-g62bl"] Oct 30 01:06:55.878762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:55.878732 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" path="/var/lib/kubelet/pods/01dc49ed-7fe2-450a-9b07-7c1e2ae56567/volumes" Oct 30 01:06:56.048576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.048534 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:56.195667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.195389 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:56.198732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.198705 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:06:56.199366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.199352 5938 cpu_manager.go:414] "containerMap: removing container" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" Oct 30 01:06:56.199417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.199368 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" Oct 30 01:06:56.199417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.199381 5938 cpu_manager.go:414] "containerMap: removing container" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:56.199417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.199386 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f03d8c-6c8b-411f-a279-71375d4597c9" containerName="istio-proxy" Oct 30 01:06:56.199516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.199510 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dc49ed-7fe2-450a-9b07-7c1e2ae56567" containerName="istio-proxy" Oct 30 01:06:56.203198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.203182 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.221466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.221444 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:06:56.228713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.228692 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-93079c\"" Oct 30 01:06:56.228828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.228737 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-93079c-dockercfg-5pw4l\"" Oct 30 01:06:56.351888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.305060 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4n9\" (UniqueName: \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.351888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.305130 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.381919 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.381121 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:06:56.396884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.394909 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:06:56.396884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.395049 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.421886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.411130 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4n9\" (UniqueName: \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.421886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.411231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.421886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.415371 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.469180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.445973 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4n9\" (UniqueName: \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\") pod \"payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.512607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.512580 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl77b\" (UniqueName: \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.512851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.512835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.516203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.516183 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:56.613784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.613746 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl77b\" (UniqueName: \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.614074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.614057 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.615054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.615035 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.636921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.636892 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl77b\" (UniqueName: \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\") pod \"payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.736085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.735949 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:56.825709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.825678 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:06:56.836134 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:06:56.836103 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69a195e_4559_41ba_9629_58f8e78bb5d7.slice/crio-66358af05209f5e80d3bcbebea8f73578700025c0a28514a81be9cb55aef79ae WatchSource:0}: Error finding container 66358af05209f5e80d3bcbebea8f73578700025c0a28514a81be9cb55aef79ae: Status 404 returned error can't find the container with id 66358af05209f5e80d3bcbebea8f73578700025c0a28514a81be9cb55aef79ae Oct 30 01:06:56.898438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:56.898365 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" event={"ID":"f69a195e-4559-41ba-9629-58f8e78bb5d7","Type":"ContainerStarted","Data":"66358af05209f5e80d3bcbebea8f73578700025c0a28514a81be9cb55aef79ae"} Oct 30 01:06:57.050533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.050511 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:06:57.051982 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:06:57.051954 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a9d2ce_671c_496e_84ec_05fef4c3d6a4.slice/crio-e66c4c552d34dbc48931ce919d84def63ef19fa976283d0a85a93b6a8b328d31 WatchSource:0}: Error finding container e66c4c552d34dbc48931ce919d84def63ef19fa976283d0a85a93b6a8b328d31: Status 404 returned error can't find the container with id e66c4c552d34dbc48931ce919d84def63ef19fa976283d0a85a93b6a8b328d31 Oct 30 01:06:57.906711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.906671 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" event={"ID":"f69a195e-4559-41ba-9629-58f8e78bb5d7","Type":"ContainerStarted","Data":"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57"} Oct 30 01:06:57.907178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.906846 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:06:57.908366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.908335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" event={"ID":"49a9d2ce-671c-496e-84ec-05fef4c3d6a4","Type":"ContainerStarted","Data":"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18"} Oct 30 01:06:57.908482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.908378 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:06:57.908482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.908395 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" event={"ID":"49a9d2ce-671c-496e-84ec-05fef4c3d6a4","Type":"ContainerStarted","Data":"e66c4c552d34dbc48931ce919d84def63ef19fa976283d0a85a93b6a8b328d31"} Oct 30 01:06:57.949162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.949121 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" podStartSLOduration=1.9491079089999999 podStartE2EDuration="1.949107909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:57.947308359 +0000 UTC m=+2070.187543474" watchObservedRunningTime="2026-10-30 01:06:57.949107909 +0000 UTC m=+2070.189343022" Oct 30 01:06:57.982989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.982964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:57.990569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:57.990532 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" podStartSLOduration=1.990522017 podStartE2EDuration="1.990522017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:57.986844888 +0000 UTC m=+2070.227080001" watchObservedRunningTime="2026-10-30 01:06:57.990522017 +0000 UTC m=+2070.230757130" Oct 30 01:06:58.835498 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:06:58.835464 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:06:59.628245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:06:59.628216 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.024263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:01.024230 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.414569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:01.414531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:03.150072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:03.150031 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:04.919393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:04.919359 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:06.599858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:06.599823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:08.384396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:08.384358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:08.918568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:08.918499 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:07:08.918568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:08.918562 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:07:10.084467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:10.084435 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:11.820352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:11.820311 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:11.967742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:11.967713 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:13.583815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:13.583767 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:13.808967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:13.808938 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:15.243573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:15.243543 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.882331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:16.882296 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:18.517352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:18.517313 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.166766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:20.166738 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:21.775924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:21.775858 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:23.614592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:23.614551 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:25.529888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:25.529237 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:27.164985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:27.164947 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.670636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.670604 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:07:28.675131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.675107 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.719405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.690558 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.719405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.690636 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.719405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.690733 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhqm\" (UniqueName: \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.719405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.690843 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.719979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.719955 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-93079c-serving-cert\"" Oct 30 01:07:28.751032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.750939 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:07:28.812054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.812017 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.812926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.812901 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.813207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.813182 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhqm\" (UniqueName: \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.841886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.813427 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.842571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.826572 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.842571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.835668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.842571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.836716 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.842571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.839336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.851575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.851549 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhqm\" (UniqueName: \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\") pod \"maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:28.874265 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:07:28.874232 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:07:28.992942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:28.983994 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:29.058209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:29.058175 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:29.446703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:29.446674 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:07:29.447556 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:07:29.447524 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb8ae34_74d6_4df6_b2e5_7180e1042032.slice/crio-b27c3b70943b860aeb1935f6405014c45be8f148846fe2e44f5001a159a8e764 WatchSource:0}: Error finding container b27c3b70943b860aeb1935f6405014c45be8f148846fe2e44f5001a159a8e764: Status 404 returned error can't find the container with id b27c3b70943b860aeb1935f6405014c45be8f148846fe2e44f5001a159a8e764 Oct 30 01:07:30.380219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.380168 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" event={"ID":"1bb8ae34-74d6-4df6-b2e5-7180e1042032","Type":"ContainerStarted","Data":"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d"} Oct 30 01:07:30.380645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.380248 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:30.380645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.380272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" event={"ID":"1bb8ae34-74d6-4df6-b2e5-7180e1042032","Type":"ContainerStarted","Data":"b27c3b70943b860aeb1935f6405014c45be8f148846fe2e44f5001a159a8e764"} Oct 30 01:07:30.425173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.425117 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" podStartSLOduration=2.052169838 podStartE2EDuration="2.425097944s" totalImagesPullingTime="372.928106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:30.420975193 +0000 UTC m=+2102.661210306" watchObservedRunningTime="2026-10-30 01:07:30.425097944 +0000 UTC m=+2102.665333060" Oct 30 01:07:30.512124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.512056 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:07:30.532745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.532719 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:07:30.732846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:30.732794 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:31.126402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.126371 5938 scope.go:122] "RemoveContainer" containerID="5546e736bc93a03bb769d91cf1fcaae39eab5f6953478c3382ee9da072ea45c5" Oct 30 01:07:31.833129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.833092 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:07:31.867707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.867667 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.889019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.888992 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:07:31.960520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960481 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.960714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960602 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.960714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960637 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48wr\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.960810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960745 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.960913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960884 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.960988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.960965 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.961160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.961140 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.961252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.961233 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:31.961318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:31.961289 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.062732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.062692 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.062797 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.062830 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48wr\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.062931 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063068 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063092 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063187 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063206 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063231 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063337 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063481 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.063878 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.063840 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.064111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.064081 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.066543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.066519 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.069222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.069199 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.086720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.086659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.086987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.086967 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48wr\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.225449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.225408 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:32.406606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.406537 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:32.630324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.630287 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:07:32.640747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.639344 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:32.640747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.639491 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:32.640747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:32.639591 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:33.415976 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:33.415931 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" event={"ID":"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b","Type":"ContainerStarted","Data":"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e"} Oct 30 01:07:33.416426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:33.416000 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" event={"ID":"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b","Type":"ContainerStarted","Data":"8c8594a51644458df2d543a511903b8c73cacfbf90ec7ad74bb2cb2e9b5c1c5c"} Oct 30 01:07:33.476958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:33.476895 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podStartSLOduration=2.476874999 podStartE2EDuration="2.476874999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:33.472468762 +0000 UTC m=+2105.712703888" watchObservedRunningTime="2026-10-30 01:07:33.476874999 +0000 UTC m=+2105.717110114" Oct 30 01:07:34.226200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:34.226160 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:34.228009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:34.227978 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:34.228173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:34.228027 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:34.323521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:34.323487 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:34.473886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:34.473669 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:35.225847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.225819 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:35.226048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.225905 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:35.361845 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.361105 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": read tcp 10.134.0.2:59842->10.134.0.233:15021: read: connection reset by peer" start-of-body= Oct 30 01:07:35.361845 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.361174 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": read tcp 10.134.0.2:59842->10.134.0.233:15021: read: connection reset by peer" Oct 30 01:07:35.436676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.436648 5938 generic.go:356] "Generic (PLEG): container finished" podID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerID="0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924" exitCode=0 Oct 30 01:07:35.436842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.436714 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerDied","Data":"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924"} Oct 30 01:07:35.437199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.437171 5938 scope.go:122] "RemoveContainer" containerID="0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924" Oct 30 01:07:35.438680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.438658 5938 generic.go:356] "Generic (PLEG): container finished" podID="77b0aee2-7805-463c-9c43-472daea37f3c" containerID="221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b" exitCode=0 Oct 30 01:07:35.438770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.438730 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerDied","Data":"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b"} Oct 30 01:07:35.439146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.439121 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:35.439210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.439187 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:35.439249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.439220 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:35.439249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.439237 5938 scope.go:122] "RemoveContainer" containerID="221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b" Oct 30 01:07:35.440958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.440934 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:35.441038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.440996 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:35.441038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:35.441029 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:07:36.226583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.226542 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:36.227146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.226605 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:36.419886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.419474 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:07:36.490962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.456650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:36.490962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.457736 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerStarted","Data":"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6"} Oct 30 01:07:36.490962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.462720 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerStarted","Data":"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008"} Oct 30 01:07:36.945930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.942986 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:07:36.967491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.943986 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:36.967491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:36.967054 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:37.178337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.178298 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:07:37.178556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.178408 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 01:07:37.178556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.178452 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 01:07:37.227948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.227310 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:37.227948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.227376 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:37.913896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.887662 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:37.913896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.887720 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:37.947166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.947128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:37.990888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:37.985710 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:07:38.177445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.177365 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 01:07:38.177445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.177413 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 01:07:38.237900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.229594 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:38.237900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.229651 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:38.337618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.337572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:38.891888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.888187 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:38.891888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:38.888256 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:39.178109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.178022 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 01:07:39.178403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.178354 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 01:07:39.228888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.228746 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 01:07:39.228888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.228803 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 01:07:39.494791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.494690 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:39.904889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.893711 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:39.904889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.893774 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:39.998887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:39.992422 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.181405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.181337 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 01:07:40.181611 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.181592 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 01:07:40.242892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.242612 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:40.244418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.244171 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:40.247644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.247625 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:07:40.885237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.885197 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:40.885849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:40.885818 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:41.178089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:41.178010 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 30 01:07:41.178089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:41.178066 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 30 01:07:41.697887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:41.697348 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:41.889899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:41.889215 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Oct 30 01:07:41.889899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:41.889272 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Oct 30 01:07:42.182087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:42.182056 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:07:42.182546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:42.182519 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:07:42.183318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:42.183295 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:07:42.890368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:42.890339 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:07:42.891377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:42.891355 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:07:43.408367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:43.408328 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:45.130534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:45.130483 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:46.996040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:46.996008 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:48.619204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:48.619170 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:48.795941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:48.780285 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:50.611012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:50.610978 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:52.338212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:52.338179 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:54.098886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:54.080213 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:55.976833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:55.976785 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:57.676446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:57.676409 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:58.835453 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:07:58.835413 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:07:59.313169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:59.313140 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:59.453673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:07:59.453638 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:01.123736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:01.123705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:02.751420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:02.751394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:04.400666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:04.400631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:06.192256 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:06.192225 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:07.829215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:07.829179 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:09.740949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:09.740912 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:10.068191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:10.068104 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.423738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:11.423698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.712005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:11.711917 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:11.961048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:11.960972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:12.485694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.485659 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:08:12.552185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.552155 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:08:12.552380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.552245 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.570759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.568519 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a5e9ab-openshift-default-dockercfg-f6wp2\"" Oct 30 01:08:12.625455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.625401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:12.626233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626205 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626254 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626303 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626328 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626359 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626450 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626512 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.626639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.626531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77lst\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.756952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.741636 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.794514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.794412 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:12.794818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.757633 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816291 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816414 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816456 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lst\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816564 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816724 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.816834 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.818420 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.818675 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.822918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.818934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.842939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.837953 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.878448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.878416 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.919885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.919547 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.962887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.959534 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:12.969031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:12.968939 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lst\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\") pod \"e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:13.176698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.176662 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:13.528212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.528183 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:08:13.529135 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:13.529104 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82864e7_24c8_41a5_b703_bdeba198cbdd.slice/crio-7be04cb4a728548807a895666c552ca2d4bfd099760a9175f95b85f0d12f45ff WatchSource:0}: Error finding container 7be04cb4a728548807a895666c552ca2d4bfd099760a9175f95b85f0d12f45ff: Status 404 returned error can't find the container with id 7be04cb4a728548807a895666c552ca2d4bfd099760a9175f95b85f0d12f45ff Oct 30 01:08:13.532138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.532100 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:13.532227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.532182 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:13.532269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.532225 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:13.598161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.598126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.924067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.924032 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerStarted","Data":"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c"} Oct 30 01:08:13.924251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.924085 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerStarted","Data":"7be04cb4a728548807a895666c552ca2d4bfd099760a9175f95b85f0d12f45ff"} Oct 30 01:08:13.968556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:13.968504 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podStartSLOduration=1.968490315 podStartE2EDuration="1.968490315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:13.966544607 +0000 UTC m=+2146.206779720" watchObservedRunningTime="2026-10-30 01:08:13.968490315 +0000 UTC m=+2146.208725434" Oct 30 01:08:14.177461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:14.177378 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:14.184786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:14.184757 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:14.930614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:14.930583 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:14.931514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:14.931495 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:08:15.067567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:15.067529 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:15.442271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:15.442235 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:17.316449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:17.316412 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:18.354617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:18.354582 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:18.493210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:18.493176 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:19.232854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:19.232820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.019404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.019373 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.166942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.166909 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:08:20.173806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.173779 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.189060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.189036 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.191136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.191112 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:08:20.203462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.203390 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-a5e9ab\"" Oct 30 01:08:20.220471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.220436 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-a5e9ab-dockercfg-sbw8b\"" Oct 30 01:08:20.431896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.374331 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xppd\" (UniqueName: \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.431896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.374509 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.474756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.474713 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:08:20.476661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.476633 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xppd\" (UniqueName: \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.476806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.476732 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.477659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.477452 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.499711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.499691 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:08:20.499931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.499790 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.500550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.500424 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xppd\" (UniqueName: \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\") pod \"payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.529933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.529911 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:20.578683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.578652 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwqv\" (UniqueName: \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.578802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.578783 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.680176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.680140 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwqv\" (UniqueName: \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.680369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.680241 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.680947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.680924 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.712209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.710858 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwqv\" (UniqueName: \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\") pod \"payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.763882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.763832 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:08:20.764305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.764276 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" podUID="1099852e-0a22-42c5-a179-cce5afeeac7c" containerName="maas-api" containerID="cri-o://cf389c09b00f6d6d1ef4007c60e97ae65a3b9802c7b03d9580e8d53fefc37010" gracePeriod=30 Oct 30 01:08:20.850382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.850357 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:20.861104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.861077 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:08:20.873176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:20.873141 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:20.875362 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:20.875335 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283de2d6_4495_4098_8291_842b85a69436.slice/crio-9a78036afdaa8d3a971b429ed31d54a6baefa5a9cb0a4a19043026ac6cc15132 WatchSource:0}: Error finding container 9a78036afdaa8d3a971b429ed31d54a6baefa5a9cb0a4a19043026ac6cc15132: Status 404 returned error can't find the container with id 9a78036afdaa8d3a971b429ed31d54a6baefa5a9cb0a4a19043026ac6cc15132 Oct 30 01:08:21.099202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.099162 5938 generic.go:356] "Generic (PLEG): container finished" podID="1099852e-0a22-42c5-a179-cce5afeeac7c" containerID="cf389c09b00f6d6d1ef4007c60e97ae65a3b9802c7b03d9580e8d53fefc37010" exitCode=0 Oct 30 01:08:21.099685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.099326 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" event={"ID":"1099852e-0a22-42c5-a179-cce5afeeac7c","Type":"ContainerDied","Data":"cf389c09b00f6d6d1ef4007c60e97ae65a3b9802c7b03d9580e8d53fefc37010"} Oct 30 01:08:21.104936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.102568 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" event={"ID":"283de2d6-4495-4098-8291-842b85a69436","Type":"ContainerStarted","Data":"9a78036afdaa8d3a971b429ed31d54a6baefa5a9cb0a4a19043026ac6cc15132"} Oct 30 01:08:21.397471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.395334 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:08:21.400678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.400023 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:08:21.409153 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:21.409115 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fb993b_8494_4f3b_8e21_5330caf854e8.slice/crio-9fa8f91d233841a6f72a9660fd404a427394e3f660ed8a4ed47754479ef051e6 WatchSource:0}: Error finding container 9fa8f91d233841a6f72a9660fd404a427394e3f660ed8a4ed47754479ef051e6: Status 404 returned error can't find the container with id 9fa8f91d233841a6f72a9660fd404a427394e3f660ed8a4ed47754479ef051e6 Oct 30 01:08:21.477469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.477413 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:21.536614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.536578 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\" (UniqueName: \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\") pod \"1099852e-0a22-42c5-a179-cce5afeeac7c\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " Oct 30 01:08:21.536794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.536676 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\") pod \"1099852e-0a22-42c5-a179-cce5afeeac7c\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " Oct 30 01:08:21.536972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.536938 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\") pod \"1099852e-0a22-42c5-a179-cce5afeeac7c\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " Oct 30 01:08:21.537362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.537330 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca" pod "1099852e-0a22-42c5-a179-cce5afeeac7c" (UID: "1099852e-0a22-42c5-a179-cce5afeeac7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:21.537450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.537396 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") pod \"1099852e-0a22-42c5-a179-cce5afeeac7c\" (UID: \"1099852e-0a22-42c5-a179-cce5afeeac7c\") " Oct 30 01:08:21.538084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.538060 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1099852e-0a22-42c5-a179-cce5afeeac7c-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:21.539241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.539196 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l" pod "1099852e-0a22-42c5-a179-cce5afeeac7c" (UID: "1099852e-0a22-42c5-a179-cce5afeeac7c"). InnerVolumeSpecName "kube-api-access-ql42l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:21.539714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.539689 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls" pod "1099852e-0a22-42c5-a179-cce5afeeac7c" (UID: "1099852e-0a22-42c5-a179-cce5afeeac7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:21.539971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.539950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs" pod "1099852e-0a22-42c5-a179-cce5afeeac7c" (UID: "1099852e-0a22-42c5-a179-cce5afeeac7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:21.640266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.640226 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:21.640266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.640266 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql42l\" (UniqueName: \"kubernetes.io/projected/1099852e-0a22-42c5-a179-cce5afeeac7c-kube-api-access-ql42l\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:21.640610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.640281 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1099852e-0a22-42c5-a179-cce5afeeac7c-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:21.656555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:21.656441 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:22.114501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.114430 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" event={"ID":"1099852e-0a22-42c5-a179-cce5afeeac7c","Type":"ContainerDied","Data":"e696de00ec3081016c018f570887b07b95768da35cf7835ba9f5ea2580b3d4d3"} Oct 30 01:08:22.115051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.114566 5938 scope.go:122] "RemoveContainer" containerID="cf389c09b00f6d6d1ef4007c60e97ae65a3b9802c7b03d9580e8d53fefc37010" Oct 30 01:08:22.115051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.114475 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5" Oct 30 01:08:22.117859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.117822 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" event={"ID":"283de2d6-4495-4098-8291-842b85a69436","Type":"ContainerStarted","Data":"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4"} Oct 30 01:08:22.118714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.118689 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:22.121058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.121024 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" event={"ID":"85fb993b-8494-4f3b-8e21-5330caf854e8","Type":"ContainerStarted","Data":"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d"} Oct 30 01:08:22.121184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.121072 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" event={"ID":"85fb993b-8494-4f3b-8e21-5330caf854e8","Type":"ContainerStarted","Data":"9fa8f91d233841a6f72a9660fd404a427394e3f660ed8a4ed47754479ef051e6"} Oct 30 01:08:22.121184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.121105 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:22.165802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.165778 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:08:22.177106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.177081 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef1084-6bc49dc4d7-lw7p5"] Oct 30 01:08:22.214148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.214107 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" podStartSLOduration=2.214095906 podStartE2EDuration="2.214095906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:22.211923881 +0000 UTC m=+2154.452158999" watchObservedRunningTime="2026-10-30 01:08:22.214095906 +0000 UTC m=+2154.454331018" Oct 30 01:08:22.256499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.256444 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" podStartSLOduration=2.256427392 podStartE2EDuration="2.256427392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:22.25309102 +0000 UTC m=+2154.493326134" watchObservedRunningTime="2026-10-30 01:08:22.256427392 +0000 UTC m=+2154.496662507" Oct 30 01:08:22.772831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.772788 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:22.860483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.860457 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:22.860615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:22.860506 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:23.130500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.130469 5938 generic.go:356] "Generic (PLEG): container finished" podID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerID="88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1" exitCode=0 Oct 30 01:08:23.130922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.130543 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerDied","Data":"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1"} Oct 30 01:08:23.131057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.131036 5938 scope.go:122] "RemoveContainer" containerID="88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1" Oct 30 01:08:23.133132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.133096 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:23.133242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.133188 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:23.133285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.133240 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:23.634460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.634430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:23.879051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:23.879021 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1099852e-0a22-42c5-a179-cce5afeeac7c" path="/var/lib/kubelet/pods/1099852e-0a22-42c5-a179-cce5afeeac7c/volumes" Oct 30 01:08:24.142315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:24.142276 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerStarted","Data":"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d"} Oct 30 01:08:24.311884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:24.311553 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:08:24.311884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:24.311762 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:24.311884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:24.311804 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:24.959828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:24.959771 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:25.314312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:25.314217 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:25.314312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:25.314280 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:25.712150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:25.712113 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:26.311370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:26.311331 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:26.311577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:26.311396 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:27.312072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.312026 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:27.312599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.312097 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:27.516431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.516397 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:08:27.517197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.517177 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1099852e-0a22-42c5-a179-cce5afeeac7c" containerName="maas-api" Oct 30 01:08:27.517197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.517195 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1099852e-0a22-42c5-a179-cce5afeeac7c" containerName="maas-api" Oct 30 01:08:27.517386 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.517354 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1099852e-0a22-42c5-a179-cce5afeeac7c" containerName="maas-api" Oct 30 01:08:27.520478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.520454 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.530711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.530691 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:08:27.535290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.535264 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-55a4c7-openshift-default-dockercfg-mbtmr\"" Oct 30 01:08:27.570233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570169 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570224 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dpn\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570355 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570400 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570631 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570681 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570765 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570795 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.570906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.570884 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672496 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672534 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672580 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672616 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672645 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dpn\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672669 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.672715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672697 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672838 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672889 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.672970 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.673042 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.673070 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.673244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.673581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.673562 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.675123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.675098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.675311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.675293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.723897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.721381 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.730055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.730016 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dpn\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\") pod \"e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:27.802422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.802330 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:27.853970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:27.852168 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:28.320684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.320597 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:28.321165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.320686 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:28.339543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.339506 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:08:28.343586 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:28.343090 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd913e5db_0733_4184_b860_83c44bb85fcc.slice/crio-a961862bc3775e48b4d4a534721e7d6a9dfde7007624c2fb8cf0aa2fde90803e WatchSource:0}: Error finding container a961862bc3775e48b4d4a534721e7d6a9dfde7007624c2fb8cf0aa2fde90803e: Status 404 returned error can't find the container with id a961862bc3775e48b4d4a534721e7d6a9dfde7007624c2fb8cf0aa2fde90803e Oct 30 01:08:28.346416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.346379 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:28.346502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.346475 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:28.346547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:28.346526 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:08:28.869609 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:08:28.869567 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:08:29.186764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.186675 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerStarted","Data":"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412"} Oct 30 01:08:29.186764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.186727 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerStarted","Data":"a961862bc3775e48b4d4a534721e7d6a9dfde7007624c2fb8cf0aa2fde90803e"} Oct 30 01:08:29.246412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.246349 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podStartSLOduration=2.246329305 podStartE2EDuration="2.246329305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:29.243021731 +0000 UTC m=+2161.483256844" watchObservedRunningTime="2026-10-30 01:08:29.246329305 +0000 UTC m=+2161.486564419" Oct 30 01:08:29.312069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.312027 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 01:08:29.312237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.312091 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 01:08:29.339271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.339232 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.539588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.539470 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.660802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.660762 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:08:29.661061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.661032 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" containerID="cri-o://10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220" gracePeriod=30 Oct 30 01:08:29.737779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.737750 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.860885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.854937 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:29.945906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:29.945103 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:30.194615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:30.194536 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:30.195354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:30.195334 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:08:30.316417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:30.316388 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:08:30.316789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:30.316767 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:08:30.317426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:30.317406 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:08:31.200947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:31.200919 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:32.842741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:32.842706 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:33.134979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:33.134954 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:08:33.628285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:33.628250 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:33.765901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:33.765839 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.145741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:34.145715 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:08:34.797604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:34.797572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.983400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:34.983373 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:08:35.028426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028455 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028487 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028519 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028547 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028644 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028721 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\") pod \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\" (UID: \"a9bcaf69-019c-426c-ac5a-a64f44ac85d5\") " Oct 30 01:08:35.028959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028823 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:35.028959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.028947 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:35.029180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029087 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:35.029180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029102 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:35.029361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029333 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:35.029657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029633 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.029730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029661 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.029730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029675 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.029730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029691 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.029730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.029705 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.030857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.030836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:35.031069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.031048 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "kube-api-access-5h48d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:35.031211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.031196 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:35.031280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.031250 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy" pod "a9bcaf69-019c-426c-ac5a-a64f44ac85d5" (UID: "a9bcaf69-019c-426c-ac5a-a64f44ac85d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:35.130951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.130922 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.130951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.130950 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h48d\" (UniqueName: \"kubernetes.io/projected/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-kube-api-access-5h48d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.131149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.130963 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.131149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.130977 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bcaf69-019c-426c-ac5a-a64f44ac85d5-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:08:35.247520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.247445 5938 generic.go:356] "Generic (PLEG): container finished" podID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerID="10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220" exitCode=0 Oct 30 01:08:35.247520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.247482 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" event={"ID":"a9bcaf69-019c-426c-ac5a-a64f44ac85d5","Type":"ContainerDied","Data":"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220"} Oct 30 01:08:35.247520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.247503 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" Oct 30 01:08:35.247520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.247520 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4" event={"ID":"a9bcaf69-019c-426c-ac5a-a64f44ac85d5","Type":"ContainerDied","Data":"6ecb8a0d84bf27abb22635aa74f8125e938e6fb974edbffc0834773a2fd4110c"} Oct 30 01:08:35.248126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.247541 5938 scope.go:122] "RemoveContainer" containerID="10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220" Oct 30 01:08:35.259602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.259575 5938 scope.go:122] "RemoveContainer" containerID="10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220" Oct 30 01:08:35.259948 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:08:35.259926 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220\": container with ID starting with 10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220 not found: ID does not exist" containerID="10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220" Oct 30 01:08:35.260047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.259955 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220"} err="failed to get container status \"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220\": rpc error: code = NotFound desc = could not find container \"10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220\": container with ID starting with 10297850dd492b64d99b9cdc132c58c8c28bfe679a34030c66a25b2815d62220 not found: ID does not exist" Oct 30 01:08:35.291018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.290984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:35.303041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.303014 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:08:35.306383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.306358 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-bh9g4"] Oct 30 01:08:35.385347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.385319 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:08:35.386287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.386265 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" Oct 30 01:08:35.386287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.386283 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" Oct 30 01:08:35.386467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.386450 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" containerName="istio-proxy" Oct 30 01:08:35.394808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.394783 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.410042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.410014 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:08:35.412264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.412145 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-55a4c7\"" Oct 30 01:08:35.425828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.425803 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:08:35.428988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.428965 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.435900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.435830 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.436019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.435939 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch46c\" (UniqueName: \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.443406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.443384 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-55a4c7-dockercfg-h7rb6\"" Oct 30 01:08:35.447282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.447263 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:08:35.469077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.469049 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:35.574937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.563983 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.574937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.564088 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmfv7\" (UniqueName: \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.574937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.564200 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.574937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.564248 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch46c\" (UniqueName: \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.574937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.565620 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.601143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.597770 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch46c\" (UniqueName: \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\") pod \"payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.678883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.677090 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.678883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.677194 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmfv7\" (UniqueName: \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.678883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.678704 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.707937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.707783 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmfv7\" (UniqueName: \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.716107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.716071 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:35.748165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.748134 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:35.880311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:35.880283 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bcaf69-019c-426c-ac5a-a64f44ac85d5" path="/var/lib/kubelet/pods/a9bcaf69-019c-426c-ac5a-a64f44ac85d5/volumes" Oct 30 01:08:36.026490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.026461 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:08:36.027318 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:36.027292 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda312df6b_443b_40e4_af94_4721b415310e.slice/crio-331a3f5ec181cc158e3bfa6d3ce223daa09901dbf1410bb4ec073d992d2902af WatchSource:0}: Error finding container 331a3f5ec181cc158e3bfa6d3ce223daa09901dbf1410bb4ec073d992d2902af: Status 404 returned error can't find the container with id 331a3f5ec181cc158e3bfa6d3ce223daa09901dbf1410bb4ec073d992d2902af Oct 30 01:08:36.066803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.066778 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:08:36.067550 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:08:36.067527 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c236db_bdc1_4dc5_813a_6222aab6f81e.slice/crio-48af93f7b13b1269c6dbb991cb7b8e833cb331c265653793a3a86c217788e5ff WatchSource:0}: Error finding container 48af93f7b13b1269c6dbb991cb7b8e833cb331c265653793a3a86c217788e5ff: Status 404 returned error can't find the container with id 48af93f7b13b1269c6dbb991cb7b8e833cb331c265653793a3a86c217788e5ff Oct 30 01:08:36.266730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.266693 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" event={"ID":"b2c236db-bdc1-4dc5-813a-6222aab6f81e","Type":"ContainerStarted","Data":"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57"} Oct 30 01:08:36.267219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.266781 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:36.267219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.266807 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" event={"ID":"b2c236db-bdc1-4dc5-813a-6222aab6f81e","Type":"ContainerStarted","Data":"48af93f7b13b1269c6dbb991cb7b8e833cb331c265653793a3a86c217788e5ff"} Oct 30 01:08:36.268914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.268879 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" event={"ID":"a312df6b-443b-40e4-af94-4721b415310e","Type":"ContainerStarted","Data":"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e"} Oct 30 01:08:36.269057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.268930 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" event={"ID":"a312df6b-443b-40e4-af94-4721b415310e","Type":"ContainerStarted","Data":"331a3f5ec181cc158e3bfa6d3ce223daa09901dbf1410bb4ec073d992d2902af"} Oct 30 01:08:36.269057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.268955 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:36.308577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.308536 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" podStartSLOduration=1.308523828 podStartE2EDuration="1.308523828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:36.305095545 +0000 UTC m=+2168.545330659" watchObservedRunningTime="2026-10-30 01:08:36.308523828 +0000 UTC m=+2168.548758941" Oct 30 01:08:36.347300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.347229 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" podStartSLOduration=1.347213955 podStartE2EDuration="1.347213955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:36.343421177 +0000 UTC m=+2168.583656292" watchObservedRunningTime="2026-10-30 01:08:36.347213955 +0000 UTC m=+2168.587449069" Oct 30 01:08:36.669838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:36.669788 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:38.425025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:38.424959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:39.619915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:39.619775 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:40.130568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:40.130535 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:41.896762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:41.896717 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:42.837885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:42.837719 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:43.017900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:43.014478 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:43.818524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:43.818480 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:45.603189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:45.602554 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:47.281478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:47.281448 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:08:47.281955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:47.281533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:08:47.395994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:47.395958 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:48.274753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:48.274703 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:49.184486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:49.184394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:50.861770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:50.861732 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:52.561998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:52.561968 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.232351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:54.232316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.391603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:54.391566 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.539062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:54.538967 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.963727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:54.963669 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:56.281531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:56.281494 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:57.964126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:57.964075 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:58.837115 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:08:58.837081 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:08:59.613486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:08:59.613441 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:01.384024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:01.383962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:03.033728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:03.033681 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:04.278335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:04.278295 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:04.593546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:04.593468 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:04.914114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:04.914075 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.124510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:05.124457 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.275245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:05.275151 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.426756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:05.426721 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.571495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:05.571399 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:05.770173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:05.770137 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:07.393096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:07.393061 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:09.256180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:09.256148 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:09.403966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:09.403934 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:11.034775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.034736 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:11.109373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.109343 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:09:11.113206 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.113190 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.131258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.131230 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:09:11.138803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.138778 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f667\"" Oct 30 01:09:11.163607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.163578 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbv8s\" (UniqueName: \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.163881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.163841 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.176467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.176445 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:09:11.179842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.179823 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.181163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.181131 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:11.194119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.194096 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f667-dockercfg-pgwp5\"" Oct 30 01:09:11.198713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.198692 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:09:11.265330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.265304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.265487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.265407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.265540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.265485 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lbz\" (UniqueName: \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.265675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.265656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbv8s\" (UniqueName: \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.266118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.266100 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.288612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.288558 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbv8s\" (UniqueName: \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\") pod \"payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.353212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.353165 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:11.375150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.375110 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.404922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.404885 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lbz\" (UniqueName: \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.415588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.376600 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.431146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.431115 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:11.454567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.450138 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lbz\" (UniqueName: \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\") pod \"payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.514836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.505092 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:11.839853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.839824 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:09:11.841140 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:09:11.841108 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a5f29_6361_44d5_aaa0_9814c3e204b1.slice/crio-9c887b74cffd9cb2e839134e7a60792fe1b48f460a6a6f889c84ced8330aecbb WatchSource:0}: Error finding container 9c887b74cffd9cb2e839134e7a60792fe1b48f460a6a6f889c84ced8330aecbb: Status 404 returned error can't find the container with id 9c887b74cffd9cb2e839134e7a60792fe1b48f460a6a6f889c84ced8330aecbb Oct 30 01:09:11.852162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:11.852136 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:09:11.856184 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:09:11.856157 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277051b9_2f54_457b_9025_c63eae5fa747.slice/crio-5cef055447a68ecfdc10cd58db355f99c926d5a05d5ae9b2d1fc26e16ad272a8 WatchSource:0}: Error finding container 5cef055447a68ecfdc10cd58db355f99c926d5a05d5ae9b2d1fc26e16ad272a8: Status 404 returned error can't find the container with id 5cef055447a68ecfdc10cd58db355f99c926d5a05d5ae9b2d1fc26e16ad272a8 Oct 30 01:09:12.610059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.610011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:12.756668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.756614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:12.770763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.770736 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" event={"ID":"280a5f29-6361-44d5-aaa0-9814c3e204b1","Type":"ContainerStarted","Data":"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1"} Oct 30 01:09:12.770984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.770792 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" event={"ID":"280a5f29-6361-44d5-aaa0-9814c3e204b1","Type":"ContainerStarted","Data":"9c887b74cffd9cb2e839134e7a60792fe1b48f460a6a6f889c84ced8330aecbb"} Oct 30 01:09:12.770984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.770895 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:12.772449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.772419 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" event={"ID":"277051b9-2f54-457b-9025-c63eae5fa747","Type":"ContainerStarted","Data":"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a"} Oct 30 01:09:12.772582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.772463 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:12.772582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.772486 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" event={"ID":"277051b9-2f54-457b-9025-c63eae5fa747","Type":"ContainerStarted","Data":"5cef055447a68ecfdc10cd58db355f99c926d5a05d5ae9b2d1fc26e16ad272a8"} Oct 30 01:09:12.809965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.809923 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" podStartSLOduration=1.8099075199999999 podStartE2EDuration="1.80990752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:12.809011807 +0000 UTC m=+2205.049246922" watchObservedRunningTime="2026-10-30 01:09:12.80990752 +0000 UTC m=+2205.050142633" Oct 30 01:09:12.858341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.858303 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" podStartSLOduration=1.858290759 podStartE2EDuration="1.858290759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:12.855256954 +0000 UTC m=+2205.095492067" watchObservedRunningTime="2026-10-30 01:09:12.858290759 +0000 UTC m=+2205.098525872" Oct 30 01:09:12.949145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.949070 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:09:12.949292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:12.949269 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" containerID="cri-o://9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe" gracePeriod=30 Oct 30 01:09:13.016395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:13.016367 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:13.762027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:13.761998 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:13.912115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:13.912082 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:14.299771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.299734 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:09:14.304484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.304460 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.323055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.323024 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:09:14.328844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.328822 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f64dc0-openshift-default-dockercfg-nvspr\"" Oct 30 01:09:14.401800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.401770 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:14.427374 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427343 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427393 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427578 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427640 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxn9\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427700 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427727 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427789 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.427920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.427902 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532386 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532455 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532564 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532651 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532719 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532826 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.532934 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.533015 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxn9\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.535886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.534274 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.555928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.551797 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:14.555928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.552567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.555928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.552842 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.555928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.553813 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.561326 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.558567 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.570845 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.570133 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.571168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.571141 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxn9\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.573670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.573639 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.583443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.583384 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\") pod \"e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:14.639947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:14.636632 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:15.002467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.002441 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:09:15.003512 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:09:15.003476 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9452a925_9485_4073_bc8c_bf81a4d79275.slice/crio-c980fb5fea58df163d98ffb08ad857d1d82e80f47614372a141950a5844f2718 WatchSource:0}: Error finding container c980fb5fea58df163d98ffb08ad857d1d82e80f47614372a141950a5844f2718: Status 404 returned error can't find the container with id c980fb5fea58df163d98ffb08ad857d1d82e80f47614372a141950a5844f2718 Oct 30 01:09:15.006442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.006405 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:15.006529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.006487 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:15.006578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.006530 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:15.495885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.492441 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:15.827130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.827047 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerStarted","Data":"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db"} Oct 30 01:09:15.827130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.827101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerStarted","Data":"c980fb5fea58df163d98ffb08ad857d1d82e80f47614372a141950a5844f2718"} Oct 30 01:09:15.873304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:15.873258 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podStartSLOduration=1.873245236 podStartE2EDuration="1.873245236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:15.869683737 +0000 UTC m=+2208.109918876" watchObservedRunningTime="2026-10-30 01:09:15.873245236 +0000 UTC m=+2208.113480346" Oct 30 01:09:16.638139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:16.638111 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:16.644263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:16.644239 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:16.834383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:16.834359 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:16.835157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:16.835134 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:09:17.206468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:17.206438 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:18.231318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.231252 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:09:18.288014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.287741 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.287903 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.287954 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288013 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288056 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288165 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288510 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.288697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.288663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\") pod \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\" (UID: \"5d49da6c-d1a1-43c5-8b3c-65f638d567e8\") " Oct 30 01:09:18.289627 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.289600 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:18.290114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.290093 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:18.290548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.290509 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:18.291333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.290763 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:18.291333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.290836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:18.292210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.292184 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:18.293057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.293033 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:18.294725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.294279 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:18.297167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.297017 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw" pod "5d49da6c-d1a1-43c5-8b3c-65f638d567e8" (UID: "5d49da6c-d1a1-43c5-8b3c-65f638d567e8"). InnerVolumeSpecName "kube-api-access-hw4mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:18.390443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390415 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390441 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390451 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390460 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390469 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4mw\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-kube-api-access-hw4mw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390477 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390485 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390492 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.390638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.390503 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d49da6c-d1a1-43c5-8b3c-65f638d567e8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:09:18.440264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.440225 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:18.590482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.590394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:18.851892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.851833 5938 generic.go:356] "Generic (PLEG): container finished" podID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerID="9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe" exitCode=0 Oct 30 01:09:18.852066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.851910 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" Oct 30 01:09:18.852066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.851926 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerDied","Data":"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe"} Oct 30 01:09:18.852066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.851997 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw" event={"ID":"5d49da6c-d1a1-43c5-8b3c-65f638d567e8","Type":"ContainerDied","Data":"ae3dde5f641ed75b45e150c048b196c5f6207fed3967687920de16c4a60f17f2"} Oct 30 01:09:18.852066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.852030 5938 scope.go:122] "RemoveContainer" containerID="9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe" Oct 30 01:09:18.864093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.864076 5938 scope.go:122] "RemoveContainer" containerID="a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5" Oct 30 01:09:18.874268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.874253 5938 scope.go:122] "RemoveContainer" containerID="9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe" Oct 30 01:09:18.874558 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:09:18.874538 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe\": container with ID starting with 9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe not found: ID does not exist" containerID="9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe" Oct 30 01:09:18.874612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.874566 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe"} err="failed to get container status \"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe\": rpc error: code = NotFound desc = could not find container \"9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe\": container with ID starting with 9aa084d19f3750ff010dc4b360c96efd311022ab825afbe331207ea85ac96ffe not found: ID does not exist" Oct 30 01:09:18.874612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.874584 5938 scope.go:122] "RemoveContainer" containerID="a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5" Oct 30 01:09:18.874886 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:09:18.874851 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5\": container with ID starting with a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5 not found: ID does not exist" containerID="a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5" Oct 30 01:09:18.874946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.874894 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5"} err="failed to get container status \"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5\": rpc error: code = NotFound desc = could not find container \"a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5\": container with ID starting with a960d2fa8e0277e3dcd0f69bfdb9f6eb104095d506c840c28768936c6f0f04e5 not found: ID does not exist" Oct 30 01:09:18.907479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.907455 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:09:18.918841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:18.918821 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19bc68-openshift-default-7b6485f77f-tz5sw"] Oct 30 01:09:19.359476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:19.354344 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:19.877196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:19.877162 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" path="/var/lib/kubelet/pods/5d49da6c-d1a1-43c5-8b3c-65f638d567e8/volumes" Oct 30 01:09:20.077255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.077223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:20.227122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.227047 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:20.353882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.353730 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:09:20.365107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.355049 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:09:20.365107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.355075 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:09:20.365107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.355120 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:09:20.365107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.355128 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:09:20.365107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.355284 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:09:20.390335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.390306 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.480999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.407714 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f64dc0\"" Oct 30 01:09:20.480999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.418895 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f64dc0-dockercfg-zr99j\"" Oct 30 01:09:20.480999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.464184 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.480999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.464262 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdb7\" (UniqueName: \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.490264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.490233 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:09:20.565687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.565653 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.565903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.565703 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdb7\" (UniqueName: \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.566348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.566329 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.587967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.587942 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdb7\" (UniqueName: \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\") pod \"payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:20.786788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:20.786723 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:21.111314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.109347 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:21.111705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.111664 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:09:21.117288 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:09:21.117242 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6823d9_fa04_412f_940e_97018cb6d90a.slice/crio-080f71010e305c3f42c01039a0b8a152ec4d4d3be797568e7d3a5263f99cddd4 WatchSource:0}: Error finding container 080f71010e305c3f42c01039a0b8a152ec4d4d3be797568e7d3a5263f99cddd4: Status 404 returned error can't find the container with id 080f71010e305c3f42c01039a0b8a152ec4d4d3be797568e7d3a5263f99cddd4 Oct 30 01:09:21.881956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.881919 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" event={"ID":"5e6823d9-fa04-412f-940e-97018cb6d90a","Type":"ContainerStarted","Data":"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e"} Oct 30 01:09:21.882332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.881980 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" event={"ID":"5e6823d9-fa04-412f-940e-97018cb6d90a","Type":"ContainerStarted","Data":"080f71010e305c3f42c01039a0b8a152ec4d4d3be797568e7d3a5263f99cddd4"} Oct 30 01:09:21.882332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.882010 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:21.922694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:21.922654 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" podStartSLOduration=1.922641648 podStartE2EDuration="1.922641648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:21.920223203 +0000 UTC m=+2214.160458341" watchObservedRunningTime="2026-10-30 01:09:21.922641648 +0000 UTC m=+2214.162876760" Oct 30 01:09:22.783083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:22.783054 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:23.813052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:23.813023 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:09:23.813436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:23.813089 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:09:24.410384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:24.410334 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:26.040988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:26.040950 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.807601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:27.807527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:28.836256 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:09:28.836222 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:09:29.489209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:29.489166 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.732472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:30.732437 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.864334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:30.864301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:31.175230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:31.175192 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:32.892968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:32.892938 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:09:32.977698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:32.977660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:35.216511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:35.216407 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:35.677451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:35.677397 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:36.078218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:36.078131 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:37.158981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:37.158929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:39.155894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:39.147656 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:39.750513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:39.750196 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:39.922526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:39.922484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:40.982347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:40.982321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:42.586529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:42.586476 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:44.259695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:44.259657 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:45.888405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:45.887365 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:45.972025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:45.971991 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:09:45.972178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:45.972077 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:46.237035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:46.236972 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:09:46.237181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:46.237038 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:09:47.331643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.331565 5938 generic.go:356] "Generic (PLEG): container finished" podID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerID="3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c" exitCode=0 Oct 30 01:09:47.333319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.331642 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerDied","Data":"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c"} Oct 30 01:09:47.333319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.332146 5938 scope.go:122] "RemoveContainer" containerID="3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c" Oct 30 01:09:47.334312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.334277 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:47.334389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.334372 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:47.334432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.334421 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:47.761468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.761438 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:47.907781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:47.907748 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:48.148727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.148698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:48.341407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.341372 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerStarted","Data":"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8"} Oct 30 01:09:48.343185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.343160 5938 generic.go:356] "Generic (PLEG): container finished" podID="d913e5db-0733-4184-b860-83c44bb85fcc" containerID="f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412" exitCode=0 Oct 30 01:09:48.343314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.343214 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerDied","Data":"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412"} Oct 30 01:09:48.343554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.343538 5938 scope.go:122] "RemoveContainer" containerID="f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412" Oct 30 01:09:48.345362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.345331 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:48.345449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.345401 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:48.345449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:48.345433 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:09:49.177661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.177566 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:09:49.178059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.178023 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 01:09:49.178220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.178092 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 01:09:49.354187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.354144 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerStarted","Data":"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3"} Oct 30 01:09:49.591937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.589423 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:49.853588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.853554 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:09:49.854406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.854334 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:49.854520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:49.854413 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:50.178535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:50.177419 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 01:09:50.178535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:50.177478 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 01:09:50.854168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:50.854137 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:50.854725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:50.854694 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:51.190823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:51.190734 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 01:09:51.217218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:51.217174 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 01:09:51.482504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:51.482418 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:51.854195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:51.854167 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:51.854612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:51.854212 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:52.177493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.177418 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 01:09:52.177493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.177472 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 01:09:52.625747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.625686 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.835731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.835687 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.853841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.853800 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:52.853989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:52.853900 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:53.177072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:53.177043 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 01:09:53.177497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:53.177110 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 01:09:53.459896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:53.459758 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:53.853305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:53.853268 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:53.853470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:53.853336 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:54.181909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.181800 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:09:54.182334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.182202 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:09:54.182826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.182805 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:09:54.823567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.823531 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 01:09:54.823810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.823775 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" containerID="cri-o://970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f" gracePeriod=30 Oct 30 01:09:54.853340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.853308 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 01:09:54.853477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:54.853362 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 01:09:55.392935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:55.392898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:55.857854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:55.857827 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:09:55.858118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:55.858097 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:09:55.858904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:55.858872 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:09:56.059545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:56.058932 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:09:56.089360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:56.089330 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:09:56.228273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:56.228202 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:56.378027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:56.377992 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:56.519004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:56.518926 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:57.135902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:57.135850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:57.282412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:57.282359 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:57.429420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:57.429337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:58.837195 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:09:58.837155 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:09:58.977700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:09:58.977658 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:00.076204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.076173 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 01:10:00.076572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.076409 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerName="discovery" containerID="cri-o://4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1" gracePeriod=30 Oct 30 01:10:00.166587 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:00.166555 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefcb6bf_7322_4db7_885d_b0f0a75866f3.slice/crio-conmon-4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefcb6bf_7322_4db7_885d_b0f0a75866f3.slice/crio-4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:10:00.183540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.183519 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 01:10:00.243284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.243461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243310 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.243461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243396 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.243679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243844 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243945 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.243991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.244024 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.244188 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.244228 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:00.244429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.244271 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\") pod \"d1d3df76-2e83-418a-ae74-51ef603bca7f\" (UID: \"d1d3df76-2e83-418a-ae74-51ef603bca7f\") " Oct 30 01:10:00.245069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.245053 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.245133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.245076 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d3df76-2e83-418a-ae74-51ef603bca7f-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.245505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.245486 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.245760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.245733 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.245961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.245940 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.246147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.246124 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:00.252048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.251747 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "kube-api-access-wh7tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:00.260120 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.260052 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.260274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.260237 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token" pod "d1d3df76-2e83-418a-ae74-51ef603bca7f" (UID: "d1d3df76-2e83-418a-ae74-51ef603bca7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348480 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348520 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348535 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348548 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348560 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d3df76-2e83-418a-ae74-51ef603bca7f-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348576 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d3df76-2e83-418a-ae74-51ef603bca7f-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.349571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.348592 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh7tf\" (UniqueName: \"kubernetes.io/projected/d1d3df76-2e83-418a-ae74-51ef603bca7f-kube-api-access-wh7tf\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.390445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.390413 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 01:10:00.449098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449064 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449186 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449317 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449357 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449467 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.449598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:00.449598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.449538 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\") pod \"defcb6bf-7322-4db7-885d-b0f0a75866f3\" (UID: \"defcb6bf-7322-4db7-885d-b0f0a75866f3\") " Oct 30 01:10:00.450322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.450293 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-ca-configmap\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.452340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.452315 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "kube-api-access-vkrd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:00.452489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.452467 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:00.452543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.452486 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:00.452749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.452731 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:00.452854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.452829 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:00.454074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.454048 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts" pod "defcb6bf-7322-4db7-885d-b0f0a75866f3" (UID: "defcb6bf-7322-4db7-885d-b0f0a75866f3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:00.513247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.513213 5938 generic.go:356] "Generic (PLEG): container finished" podID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerID="970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f" exitCode=0 Oct 30 01:10:00.513403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.513256 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" Oct 30 01:10:00.513403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.513284 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" event={"ID":"d1d3df76-2e83-418a-ae74-51ef603bca7f","Type":"ContainerDied","Data":"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f"} Oct 30 01:10:00.513403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.513341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb" event={"ID":"d1d3df76-2e83-418a-ae74-51ef603bca7f","Type":"ContainerDied","Data":"fde88e0f80d65db6b53ebf738551a5b36ce1386451c34bfaab57877ec205a060"} Oct 30 01:10:00.513403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.513363 5938 scope.go:122] "RemoveContainer" containerID="970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f" Oct 30 01:10:00.515274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.515249 5938 generic.go:356] "Generic (PLEG): container finished" podID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerID="4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1" exitCode=0 Oct 30 01:10:00.515411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.515286 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" event={"ID":"defcb6bf-7322-4db7-885d-b0f0a75866f3","Type":"ContainerDied","Data":"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1"} Oct 30 01:10:00.515411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.515316 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" Oct 30 01:10:00.515411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.515332 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq" event={"ID":"defcb6bf-7322-4db7-885d-b0f0a75866f3","Type":"ContainerDied","Data":"c929fd4a8213ea020954f3fadcc6db5a9871565e6ca1baee390257f4a1890f0f"} Oct 30 01:10:00.526562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.526543 5938 scope.go:122] "RemoveContainer" containerID="970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f" Oct 30 01:10:00.526992 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:00.526950 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f\": container with ID starting with 970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f not found: ID does not exist" containerID="970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f" Oct 30 01:10:00.527104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.527002 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f"} err="failed to get container status \"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f\": rpc error: code = NotFound desc = could not find container \"970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f\": container with ID starting with 970e56d5d07886055f6aeea08fb6ec8a8619e947335e330d7e5d18a205d14a0f not found: ID does not exist" Oct 30 01:10:00.527104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.527026 5938 scope.go:122] "RemoveContainer" containerID="4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1" Oct 30 01:10:00.537788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.537770 5938 scope.go:122] "RemoveContainer" containerID="4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1" Oct 30 01:10:00.538088 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:00.538072 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1\": container with ID starting with 4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1 not found: ID does not exist" containerID="4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1" Oct 30 01:10:00.538142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.538095 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1"} err="failed to get container status \"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1\": rpc error: code = NotFound desc = could not find container \"4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1\": container with ID starting with 4c68d755c0902b452007207d0f4eefbc88044315ecb0759898a1e7f43c428ff1 not found: ID does not exist" Oct 30 01:10:00.553724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553697 5938 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-csr-dns-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.553724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553722 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrd7\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-kube-api-access-vkrd7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.553724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553733 5938 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-cacerts\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.554181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553742 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.554181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553756 5938 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/defcb6bf-7322-4db7-885d-b0f0a75866f3-istio-kubeconfig\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.554181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.553768 5938 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/defcb6bf-7322-4db7-885d-b0f0a75866f3-local-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:00.565706 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.565678 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 01:10:00.569109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.569082 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5gbdq"] Oct 30 01:10:00.600694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.600672 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 01:10:00.605020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.604998 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-5qfmb"] Oct 30 01:10:00.904209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:00.904176 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:01.060212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.060172 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" podUID="a312df6b-443b-40e4-af94-4721b415310e" containerName="payload-processing" containerID="cri-o://7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e" gracePeriod=25 Oct 30 01:10:01.089794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.089758 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" podUID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" containerName="payload-pre-processing" containerID="cri-o://e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57" gracePeriod=25 Oct 30 01:10:01.379283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.379255 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:10:01.399296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.399247 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:10:01.468269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.468242 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\") pod \"a312df6b-443b-40e4-af94-4721b415310e\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " Oct 30 01:10:01.468502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.468479 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\") pod \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " Oct 30 01:10:01.468559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.468532 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\" (UniqueName: \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\") pod \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\" (UID: \"b2c236db-bdc1-4dc5-813a-6222aab6f81e\") " Oct 30 01:10:01.468661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.468641 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume" pod "a312df6b-443b-40e4-af94-4721b415310e" (UID: "a312df6b-443b-40e4-af94-4721b415310e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:01.468661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.468652 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\" (UniqueName: \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\") pod \"a312df6b-443b-40e4-af94-4721b415310e\" (UID: \"a312df6b-443b-40e4-af94-4721b415310e\") " Oct 30 01:10:01.469144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.469111 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume" pod "b2c236db-bdc1-4dc5-813a-6222aab6f81e" (UID: "b2c236db-bdc1-4dc5-813a-6222aab6f81e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:01.469371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.469343 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a312df6b-443b-40e4-af94-4721b415310e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:01.471215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.471196 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7" pod "b2c236db-bdc1-4dc5-813a-6222aab6f81e" (UID: "b2c236db-bdc1-4dc5-813a-6222aab6f81e"). InnerVolumeSpecName "kube-api-access-xmfv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:01.471328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.471225 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c" pod "a312df6b-443b-40e4-af94-4721b415310e" (UID: "a312df6b-443b-40e4-af94-4721b415310e"). InnerVolumeSpecName "kube-api-access-ch46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:01.527056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.527018 5938 generic.go:356] "Generic (PLEG): container finished" podID="a312df6b-443b-40e4-af94-4721b415310e" containerID="7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e" exitCode=0 Oct 30 01:10:01.527191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.527081 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" Oct 30 01:10:01.527191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.527109 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" event={"ID":"a312df6b-443b-40e4-af94-4721b415310e","Type":"ContainerDied","Data":"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e"} Oct 30 01:10:01.527191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.527186 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h" event={"ID":"a312df6b-443b-40e4-af94-4721b415310e","Type":"ContainerDied","Data":"331a3f5ec181cc158e3bfa6d3ce223daa09901dbf1410bb4ec073d992d2902af"} Oct 30 01:10:01.527343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.527211 5938 scope.go:122] "RemoveContainer" containerID="7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e" Oct 30 01:10:01.528713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.528686 5938 generic.go:356] "Generic (PLEG): container finished" podID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" containerID="e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57" exitCode=0 Oct 30 01:10:01.528842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.528734 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" Oct 30 01:10:01.528842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.528776 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" event={"ID":"b2c236db-bdc1-4dc5-813a-6222aab6f81e","Type":"ContainerDied","Data":"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57"} Oct 30 01:10:01.528842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.528822 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6" event={"ID":"b2c236db-bdc1-4dc5-813a-6222aab6f81e","Type":"ContainerDied","Data":"48af93f7b13b1269c6dbb991cb7b8e833cb331c265653793a3a86c217788e5ff"} Oct 30 01:10:01.538643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.538600 5938 scope.go:122] "RemoveContainer" containerID="7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e" Oct 30 01:10:01.539007 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:01.538979 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e\": container with ID starting with 7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e not found: ID does not exist" containerID="7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e" Oct 30 01:10:01.539113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.539014 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e"} err="failed to get container status \"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e\": rpc error: code = NotFound desc = could not find container \"7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e\": container with ID starting with 7a1d93e657acab4bdfd25136de3abe65a5e5ee4d2ddf1198176e7e4a4b6b4d3e not found: ID does not exist" Oct 30 01:10:01.539113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.539031 5938 scope.go:122] "RemoveContainer" containerID="e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57" Oct 30 01:10:01.548892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.548877 5938 scope.go:122] "RemoveContainer" containerID="e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57" Oct 30 01:10:01.549144 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:01.549127 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57\": container with ID starting with e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57 not found: ID does not exist" containerID="e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57" Oct 30 01:10:01.549199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.549149 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57"} err="failed to get container status \"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57\": rpc error: code = NotFound desc = could not find container \"e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57\": container with ID starting with e28070cc3c8e07919356d97abc8d74ac45836fd78805b389d3f9a32f098dcc57 not found: ID does not exist" Oct 30 01:10:01.570926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.570902 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c236db-bdc1-4dc5-813a-6222aab6f81e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:01.570926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.570925 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmfv7\" (UniqueName: \"kubernetes.io/projected/b2c236db-bdc1-4dc5-813a-6222aab6f81e-kube-api-access-xmfv7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:01.571055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.570939 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch46c\" (UniqueName: \"kubernetes.io/projected/a312df6b-443b-40e4-af94-4721b415310e-kube-api-access-ch46c\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:01.580542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.580492 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:10:01.585143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.585124 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-d778575bc-rn5m6"] Oct 30 01:10:01.619968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.619948 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:10:01.624934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.624914 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-9c65778db-9wf5h"] Oct 30 01:10:01.879100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.879073 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a312df6b-443b-40e4-af94-4721b415310e" path="/var/lib/kubelet/pods/a312df6b-443b-40e4-af94-4721b415310e/volumes" Oct 30 01:10:01.879472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.879458 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" path="/var/lib/kubelet/pods/b2c236db-bdc1-4dc5-813a-6222aab6f81e/volumes" Oct 30 01:10:01.879926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.879912 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" path="/var/lib/kubelet/pods/d1d3df76-2e83-418a-ae74-51ef603bca7f/volumes" Oct 30 01:10:01.880671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:01.880656 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" path="/var/lib/kubelet/pods/defcb6bf-7322-4db7-885d-b0f0a75866f3/volumes" Oct 30 01:10:02.511209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:02.511181 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:02.570723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:02.570257 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 01:10:02.570723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:02.570557 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" containerID="cri-o://d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0" gracePeriod=30 Oct 30 01:10:04.208211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:04.208177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:05.898211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:05.898180 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.762152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.762051 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:07.861114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.861090 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 01:10:07.949582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949555 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.949789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949609 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.949789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949668 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.949789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949721 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.949789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.950059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949819 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.950059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949881 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:07.950059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949927 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.950059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.949976 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.950284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.950105 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:07.950284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.950120 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:07.950284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.950127 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\") pod \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\" (UID: \"4b9f66cf-00e0-47ee-9431-8763de57c1b4\") " Oct 30 01:10:07.950453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.950300 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:07.950453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.950357 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:07.951083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.951054 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:07.951214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.951085 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:07.951214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.951102 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:07.951214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.951117 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:07.951214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.951132 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:07.952208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.952182 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "kube-api-access-fb9t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:07.952304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.952224 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:07.952408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.952390 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:07.952939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:07.952917 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo" pod "4b9f66cf-00e0-47ee-9431-8763de57c1b4" (UID: "4b9f66cf-00e0-47ee-9431-8763de57c1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:08.052445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.052405 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:08.052445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.052439 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:08.052639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.052455 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb9t5\" (UniqueName: \"kubernetes.io/projected/4b9f66cf-00e0-47ee-9431-8763de57c1b4-kube-api-access-fb9t5\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:08.052639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.052468 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f66cf-00e0-47ee-9431-8763de57c1b4-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:08.598302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.598265 5938 generic.go:356] "Generic (PLEG): container finished" podID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerID="d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0" exitCode=0 Oct 30 01:10:08.598493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.598313 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" Oct 30 01:10:08.598493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.598344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" event={"ID":"4b9f66cf-00e0-47ee-9431-8763de57c1b4","Type":"ContainerDied","Data":"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0"} Oct 30 01:10:08.598493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.598399 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph" event={"ID":"4b9f66cf-00e0-47ee-9431-8763de57c1b4","Type":"ContainerDied","Data":"7fb2b880ed05e784b5fa24b0c77b4491d509b49801e0b660daa817b4a9e0b9ef"} Oct 30 01:10:08.598493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.598422 5938 scope.go:122] "RemoveContainer" containerID="d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0" Oct 30 01:10:08.609638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.609609 5938 scope.go:122] "RemoveContainer" containerID="d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0" Oct 30 01:10:08.609895 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:08.609875 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0\": container with ID starting with d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0 not found: ID does not exist" containerID="d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0" Oct 30 01:10:08.609965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.609903 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0"} err="failed to get container status \"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0\": rpc error: code = NotFound desc = could not find container \"d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0\": container with ID starting with d10db3bf798c71482c7d9a8271940fc6a9f05bf168d7571ca050bfaf95681ac0 not found: ID does not exist" Oct 30 01:10:08.661409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.661385 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 01:10:08.673231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:08.673212 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kxdph"] Oct 30 01:10:09.382439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:09.382361 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:09.880708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:09.880673 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" path="/var/lib/kubelet/pods/4b9f66cf-00e0-47ee-9431-8763de57c1b4/volumes" Oct 30 01:10:09.891171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:09.891138 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:11.066201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:11.066163 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:11.248431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:11.248398 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:13.287085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:13.287058 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.024391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:15.024342 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:16.769061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:16.769017 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:18.584139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:18.584107 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:20.269303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:20.269266 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:22.305564 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:22.305527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.063720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:24.063687 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:25.708918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:25.703346 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:27.390772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:27.390724 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:28.837315 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:28.837278 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:10:29.200489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:29.200449 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:30.854788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.854763 5938 generic.go:356] "Generic (PLEG): container finished" podID="9452a925-9485-4073-bc8c-bf81a4d79275" containerID="e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db" exitCode=0 Oct 30 01:10:30.855179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.854834 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerDied","Data":"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db"} Oct 30 01:10:30.855391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.855373 5938 scope.go:122] "RemoveContainer" containerID="e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db" Oct 30 01:10:30.857116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.857089 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:30.857197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.857159 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:30.857240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.857195 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:30.985482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:30.985444 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:31.834907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:31.834848 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:10:31.864041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:31.864012 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerStarted","Data":"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00"} Oct 30 01:10:32.335110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.335074 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:32.544185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.544147 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:32.637547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.637514 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:10:32.638741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.638712 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:32.638926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.638758 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:32.733888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.723648 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.745989 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" containerName="payload-pre-processing" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746020 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" containerName="payload-pre-processing" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746038 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746045 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746076 5938 cpu_manager.go:414] "containerMap: removing container" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerName="discovery" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746083 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerName="discovery" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746148 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a312df6b-443b-40e4-af94-4721b415310e" containerName="payload-processing" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746156 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a312df6b-443b-40e4-af94-4721b415310e" containerName="payload-processing" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746187 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746195 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746407 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9f66cf-00e0-47ee-9431-8763de57c1b4" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746426 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c236db-bdc1-4dc5-813a-6222aab6f81e" containerName="payload-pre-processing" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746436 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d3df76-2e83-418a-ae74-51ef603bca7f" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746450 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="defcb6bf-7322-4db7-885d-b0f0a75866f3" containerName="discovery" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746465 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d49da6c-d1a1-43c5-8b3c-65f638d567e8" containerName="istio-proxy" Oct 30 01:10:32.749119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.746474 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a312df6b-443b-40e4-af94-4721b415310e" containerName="payload-processing" Oct 30 01:10:32.774884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.771068 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:10:32.774884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.771210 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:32.815884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.810066 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-55a4c7-dockercfg-c6klm\"" Oct 30 01:10:32.928385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.911082 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:10:32.938460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.938423 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-55a4c7\"" Oct 30 01:10:32.944243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.944214 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:10:32.944415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:32.944298 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.015754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.015706 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.016010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.015993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvr9\" (UniqueName: \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.117614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.117580 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvr9\" (UniqueName: \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.117798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.117633 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.117915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.117854 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.118009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.117941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8g92\" (UniqueName: \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.118608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.118580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.140424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.140395 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvr9\" (UniqueName: \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\") pod \"payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.219121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.219035 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.219300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.219181 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8g92\" (UniqueName: \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.219792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.219765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.239624 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.239604 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:33.241798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.241776 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8g92\" (UniqueName: \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\") pod \"payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.262205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.262183 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:33.509180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.509089 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:33.615717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.615640 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:10:33.617085 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:33.617054 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a8fbce_e322_4ccf_877c_e11bdd79d2cf.slice/crio-18f6e6561eb9d82b52391010edc2f4f04d2cc508d8b74e18c45a80bfc4e917c3 WatchSource:0}: Error finding container 18f6e6561eb9d82b52391010edc2f4f04d2cc508d8b74e18c45a80bfc4e917c3: Status 404 returned error can't find the container with id 18f6e6561eb9d82b52391010edc2f4f04d2cc508d8b74e18c45a80bfc4e917c3 Oct 30 01:10:33.619307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.619285 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:10:33.620764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.620655 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:10:33.621305 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:33.621280 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4fad2d_de82_48f8_8de1_23a1b11414fb.slice/crio-51ce7cee2c5731732d139200df958462cdad840ceef243a21bcded405979382f WatchSource:0}: Error finding container 51ce7cee2c5731732d139200df958462cdad840ceef243a21bcded405979382f: Status 404 returned error can't find the container with id 51ce7cee2c5731732d139200df958462cdad840ceef243a21bcded405979382f Oct 30 01:10:33.637893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.637842 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:33.637981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.637889 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:33.937139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.937105 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" event={"ID":"7c4fad2d-de82-48f8-8de1-23a1b11414fb","Type":"ContainerStarted","Data":"51ce7cee2c5731732d139200df958462cdad840ceef243a21bcded405979382f"} Oct 30 01:10:33.938629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:33.938602 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" event={"ID":"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf","Type":"ContainerStarted","Data":"18f6e6561eb9d82b52391010edc2f4f04d2cc508d8b74e18c45a80bfc4e917c3"} Oct 30 01:10:34.637692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:34.637655 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:34.637998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:34.637714 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:35.225753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:35.225675 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:35.637373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:35.637336 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:35.637574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:35.637392 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:36.321348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.321319 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:10:36.350070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.347611 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:10:36.488888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.484142 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:36.630310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.630272 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:36.637463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.637436 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:36.637649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.637486 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:36.774813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:36.774777 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:37.068949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.063288 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" event={"ID":"7c4fad2d-de82-48f8-8de1-23a1b11414fb","Type":"ContainerStarted","Data":"360ad671ef1ddd38e003c566f092b74a51f00c66943b72aad805e2c0246d1ffc"} Oct 30 01:10:37.068949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.064483 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:37.068949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.066621 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" event={"ID":"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf","Type":"ContainerStarted","Data":"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16"} Oct 30 01:10:37.068949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.067297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:37.120889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.120123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:37.131885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.129651 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" podStartSLOduration=2.429970061 podStartE2EDuration="5.129633227s" totalImagesPullingTime="2.699663166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:37.110632598 +0000 UTC m=+2289.350867712" watchObservedRunningTime="2026-10-30 01:10:37.129633227 +0000 UTC m=+2289.369868341" Oct 30 01:10:37.156224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.156174 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" podStartSLOduration=2.518222022 podStartE2EDuration="5.156161739s" totalImagesPullingTime="2.637939717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:37.15222283 +0000 UTC m=+2289.392457944" watchObservedRunningTime="2026-10-30 01:10:37.156161739 +0000 UTC m=+2289.396396865" Oct 30 01:10:37.371886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.371544 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:37.638416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.638241 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 01:10:37.638416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:37.638297 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 01:10:38.642127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:38.642098 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:10:38.642611 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:38.642592 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:10:38.643101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:38.643074 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:38.643227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:38.643216 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:10:38.832940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:38.832904 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:39.954747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:39.954715 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.424106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.424056 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:10:40.485021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.482482 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:10:40.485021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.482576 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.497978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.497942 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"default-dockercfg-sgnxq\"" Oct 30 01:10:40.508791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.508760 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"sub-test-model-beeba4-kserve-self-signed-certs\"" Oct 30 01:10:40.508975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.508816 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"openshift-service-ca.crt\"" Oct 30 01:10:40.509125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.509107 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"kube-root-ca.crt\"" Oct 30 01:10:40.626422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.626382 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.666304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666280 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.666442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666421 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.666501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666488 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cst\" (UniqueName: \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.666545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666518 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.666545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666537 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.666616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.666557 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.767971 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768088 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cst\" (UniqueName: \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768151 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768184 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768214 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768389 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.768917 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.769663 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.777961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.770222 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.803940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.792535 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.803940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.801922 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.807885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.805411 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:40.808610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:40.808580 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cst\" (UniqueName: \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\") pod \"sub-test-model-beeba4-kserve-7995dd78c9-tb68n\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:41.104725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.104700 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:41.322405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.322353 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" podUID="280a5f29-6361-44d5-aaa0-9814c3e204b1" containerName="payload-processing" containerID="cri-o://26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1" gracePeriod=25 Oct 30 01:10:41.348159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.348126 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" podUID="277051b9-2f54-457b-9025-c63eae5fa747" containerName="payload-pre-processing" containerID="cri-o://b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a" gracePeriod=25 Oct 30 01:10:41.408902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.408879 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:10:41.426219 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:41.426165 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a80e72_6893_4bb6_a694_1c06cff37416.slice/crio-a816cf72604ef850e3e981f99edac62a79f2365099d5cf3d5db3549214d82265 WatchSource:0}: Error finding container a816cf72604ef850e3e981f99edac62a79f2365099d5cf3d5db3549214d82265: Status 404 returned error can't find the container with id a816cf72604ef850e3e981f99edac62a79f2365099d5cf3d5db3549214d82265 Oct 30 01:10:41.623398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.623366 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:10:41.679414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.679388 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:10:41.784457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784238 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\" (UniqueName: \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\") pod \"280a5f29-6361-44d5-aaa0-9814c3e204b1\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " Oct 30 01:10:41.784457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784275 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\") pod \"280a5f29-6361-44d5-aaa0-9814c3e204b1\" (UID: \"280a5f29-6361-44d5-aaa0-9814c3e204b1\") " Oct 30 01:10:41.784457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784329 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\") pod \"277051b9-2f54-457b-9025-c63eae5fa747\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " Oct 30 01:10:41.784457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784372 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\" (UniqueName: \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\") pod \"277051b9-2f54-457b-9025-c63eae5fa747\" (UID: \"277051b9-2f54-457b-9025-c63eae5fa747\") " Oct 30 01:10:41.784934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784760 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume" pod "277051b9-2f54-457b-9025-c63eae5fa747" (UID: "277051b9-2f54-457b-9025-c63eae5fa747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:41.784934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.784778 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume" pod "280a5f29-6361-44d5-aaa0-9814c3e204b1" (UID: "280a5f29-6361-44d5-aaa0-9814c3e204b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:41.785252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.785172 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277051b9-2f54-457b-9025-c63eae5fa747-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:41.785252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.785195 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280a5f29-6361-44d5-aaa0-9814c3e204b1-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:41.786606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.786584 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s" pod "280a5f29-6361-44d5-aaa0-9814c3e204b1" (UID: "280a5f29-6361-44d5-aaa0-9814c3e204b1"). InnerVolumeSpecName "kube-api-access-sbv8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:41.786998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.786847 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz" pod "277051b9-2f54-457b-9025-c63eae5fa747" (UID: "277051b9-2f54-457b-9025-c63eae5fa747"). InnerVolumeSpecName "kube-api-access-f7lbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:41.886389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.886363 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lbz\" (UniqueName: \"kubernetes.io/projected/277051b9-2f54-457b-9025-c63eae5fa747-kube-api-access-f7lbz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:41.886389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:41.886390 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbv8s\" (UniqueName: \"kubernetes.io/projected/280a5f29-6361-44d5-aaa0-9814c3e204b1-kube-api-access-sbv8s\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:10:42.110977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.110947 5938 generic.go:356] "Generic (PLEG): container finished" podID="280a5f29-6361-44d5-aaa0-9814c3e204b1" containerID="26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1" exitCode=0 Oct 30 01:10:42.111311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.110996 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" Oct 30 01:10:42.111311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.111031 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" event={"ID":"280a5f29-6361-44d5-aaa0-9814c3e204b1","Type":"ContainerDied","Data":"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1"} Oct 30 01:10:42.111311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.111092 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq" event={"ID":"280a5f29-6361-44d5-aaa0-9814c3e204b1","Type":"ContainerDied","Data":"9c887b74cffd9cb2e839134e7a60792fe1b48f460a6a6f889c84ced8330aecbb"} Oct 30 01:10:42.111311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.111142 5938 scope.go:122] "RemoveContainer" containerID="26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1" Oct 30 01:10:42.113093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.113067 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" event={"ID":"42a80e72-6893-4bb6-a694-1c06cff37416","Type":"ContainerStarted","Data":"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e"} Oct 30 01:10:42.113214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.113104 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" event={"ID":"42a80e72-6893-4bb6-a694-1c06cff37416","Type":"ContainerStarted","Data":"a816cf72604ef850e3e981f99edac62a79f2365099d5cf3d5db3549214d82265"} Oct 30 01:10:42.116944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.116844 5938 generic.go:356] "Generic (PLEG): container finished" podID="277051b9-2f54-457b-9025-c63eae5fa747" containerID="b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a" exitCode=0 Oct 30 01:10:42.116944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.116932 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" Oct 30 01:10:42.117100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.116931 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" event={"ID":"277051b9-2f54-457b-9025-c63eae5fa747","Type":"ContainerDied","Data":"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a"} Oct 30 01:10:42.117100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.117045 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j" event={"ID":"277051b9-2f54-457b-9025-c63eae5fa747","Type":"ContainerDied","Data":"5cef055447a68ecfdc10cd58db355f99c926d5a05d5ae9b2d1fc26e16ad272a8"} Oct 30 01:10:42.122228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.122213 5938 scope.go:122] "RemoveContainer" containerID="26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1" Oct 30 01:10:42.122486 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:42.122464 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1\": container with ID starting with 26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1 not found: ID does not exist" containerID="26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1" Oct 30 01:10:42.122539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.122496 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1"} err="failed to get container status \"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1\": rpc error: code = NotFound desc = could not find container \"26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1\": container with ID starting with 26aa110c95f1863dc65bd41eb25eb8fffd0ae120218f6fc22d4b11b486fd8fe1 not found: ID does not exist" Oct 30 01:10:42.122539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.122516 5938 scope.go:122] "RemoveContainer" containerID="b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a" Oct 30 01:10:42.132325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.132307 5938 scope.go:122] "RemoveContainer" containerID="b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a" Oct 30 01:10:42.132585 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:42.132566 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a\": container with ID starting with b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a not found: ID does not exist" containerID="b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a" Oct 30 01:10:42.132671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.132590 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a"} err="failed to get container status \"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a\": rpc error: code = NotFound desc = could not find container \"b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a\": container with ID starting with b0d0ca702972a5ad4f45d814fda7f6266e1608481dd0f31e2df8a7c74c33299a not found: ID does not exist" Oct 30 01:10:42.167519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.167494 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:10:42.177552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.177531 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-64d5696dc4-hddsq"] Oct 30 01:10:42.204389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.204349 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" podStartSLOduration=2.20433867 podStartE2EDuration="2.20433867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:42.20128812 +0000 UTC m=+2294.441523232" watchObservedRunningTime="2026-10-30 01:10:42.20433867 +0000 UTC m=+2294.444573783" Oct 30 01:10:42.237441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.237419 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:10:42.241250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.241229 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-5f44887555-s9n2j"] Oct 30 01:10:42.412978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:42.412887 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:43.105497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.105471 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:43.421707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.421638 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:10:43.422715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422692 5938 cpu_manager.go:414] "containerMap: removing container" podUID="280a5f29-6361-44d5-aaa0-9814c3e204b1" containerName="payload-processing" Oct 30 01:10:43.422766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422719 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a5f29-6361-44d5-aaa0-9814c3e204b1" containerName="payload-processing" Oct 30 01:10:43.422766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422753 5938 cpu_manager.go:414] "containerMap: removing container" podUID="277051b9-2f54-457b-9025-c63eae5fa747" containerName="payload-pre-processing" Oct 30 01:10:43.422766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422761 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277051b9-2f54-457b-9025-c63eae5fa747" containerName="payload-pre-processing" Oct 30 01:10:43.422967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422954 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a5f29-6361-44d5-aaa0-9814c3e204b1" containerName="payload-processing" Oct 30 01:10:43.423011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.422977 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="277051b9-2f54-457b-9025-c63eae5fa747" containerName="payload-pre-processing" Oct 30 01:10:43.427055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.427034 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.442565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.442533 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:10:43.451204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.451166 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a5e9ab\"/\"openshift-service-ca.crt\"" Oct 30 01:10:43.451372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.451346 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-a5e9ab\"/\"kube-root-ca.crt\"" Oct 30 01:10:43.454730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.454705 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a5e9ab\"/\"default-dockercfg-pkdbb\"" Oct 30 01:10:43.454855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.454751 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-a5e9ab\"/\"auth-test-model-a5e9ab-kserve-self-signed-certs\"" Oct 30 01:10:43.516001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.515974 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:43.607683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607652 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxhz\" (UniqueName: \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.607858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607780 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.607858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607820 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.607858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.608023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607921 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.608023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.607943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708759 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxhz\" (UniqueName: \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708827 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708850 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708904 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708932 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.708972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.708952 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.709259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.709226 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.709325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.709306 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.709440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.709359 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.711193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.711165 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.711352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.711335 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.749935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.734915 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxhz\" (UniqueName: \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\") pod \"auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.759191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.759154 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:43.886520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.886485 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277051b9-2f54-457b-9025-c63eae5fa747" path="/var/lib/kubelet/pods/277051b9-2f54-457b-9025-c63eae5fa747/volumes" Oct 30 01:10:43.887299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:43.887277 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280a5f29-6361-44d5-aaa0-9814c3e204b1" path="/var/lib/kubelet/pods/280a5f29-6361-44d5-aaa0-9814c3e204b1/volumes" Oct 30 01:10:44.120082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:44.120056 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:10:44.121426 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:44.121402 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2779dc_475b_45cc_a8c9_c282b24adeb1.slice/crio-f410039d6e12e5695af5cab786959f7b89e760f9b3e7796f1fbf854eb4bf33a2 WatchSource:0}: Error finding container f410039d6e12e5695af5cab786959f7b89e760f9b3e7796f1fbf854eb4bf33a2: Status 404 returned error can't find the container with id f410039d6e12e5695af5cab786959f7b89e760f9b3e7796f1fbf854eb4bf33a2 Oct 30 01:10:44.135675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:44.135648 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" event={"ID":"bf2779dc-475b-45cc-a8c9-c282b24adeb1","Type":"ContainerStarted","Data":"f410039d6e12e5695af5cab786959f7b89e760f9b3e7796f1fbf854eb4bf33a2"} Oct 30 01:10:44.301730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:44.301695 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:45.144432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:45.144392 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" event={"ID":"bf2779dc-475b-45cc-a8c9-c282b24adeb1","Type":"ContainerStarted","Data":"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b"} Oct 30 01:10:45.186279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:45.186231 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" podStartSLOduration=2.186218708 podStartE2EDuration="2.186218708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:45.183642118 +0000 UTC m=+2297.423877241" watchObservedRunningTime="2026-10-30 01:10:45.186218708 +0000 UTC m=+2297.426453824" Oct 30 01:10:45.615126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:45.615089 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:45.759973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:45.759942 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:46.040818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.040744 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:46.199837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.199806 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:46.567509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.567465 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:10:46.572356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.572327 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.596967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.596940 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a3619c-openshift-default-dockercfg-4lvjq\"" Oct 30 01:10:46.613022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.612998 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:10:46.653056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.653025 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:46.670434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670412 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5g2\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670445 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670464 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670485 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670646 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670711 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670778 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670798 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.670978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.670961 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.796998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.796954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5g2\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.807733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.807702 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.808462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.808439 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.808646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.808629 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.808801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.808781 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.808987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.808970 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.809162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.809147 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.809273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.809260 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.809516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.809500 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.810066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.810037 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.810242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.799247 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:46.810676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.810655 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.814880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.811052 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.819001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.815314 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.822469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.821626 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.848891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.832856 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.848891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.847792 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.848891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.848157 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.849946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.849918 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5g2\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\") pod \"e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:46.891885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:46.891091 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:47.250089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:47.250064 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:10:47.251098 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:47.251078 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd4836f_89b7_4245_8394_35dca2763ecd.slice/crio-fc2906e145ffd1e1f8295e041ede2ed9e73a2c66e1a33adc15e6e891c4c18a75 WatchSource:0}: Error finding container fc2906e145ffd1e1f8295e041ede2ed9e73a2c66e1a33adc15e6e891c4c18a75: Status 404 returned error can't find the container with id fc2906e145ffd1e1f8295e041ede2ed9e73a2c66e1a33adc15e6e891c4c18a75 Oct 30 01:10:47.254052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:47.254011 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:47.254152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:47.254100 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:47.254152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:47.254142 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:47.850005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:47.849971 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:48.188065 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.188027 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerStarted","Data":"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b"} Oct 30 01:10:48.188271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.188095 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerStarted","Data":"fc2906e145ffd1e1f8295e041ede2ed9e73a2c66e1a33adc15e6e891c4c18a75"} Oct 30 01:10:48.246969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.246923 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podStartSLOduration=2.246909056 podStartE2EDuration="2.246909056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:48.242673128 +0000 UTC m=+2300.482908244" watchObservedRunningTime="2026-10-30 01:10:48.246909056 +0000 UTC m=+2300.487144168" Oct 30 01:10:48.570036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.569956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:48.895966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.895942 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:48.902249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:48.902226 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:49.087816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.087780 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:10:49.088032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.087848 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:10:49.197130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.197042 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:49.198039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.198017 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:10:49.227367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.227332 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.619606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.619573 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:10:49.624816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.624793 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.640159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.640135 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a7ca1f-openshift-default-dockercfg-774lk\"" Oct 30 01:10:49.646834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.646814 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:10:49.672793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.672766 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.672998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.672968 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673113 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673193 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblth\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673243 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673308 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673384 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673428 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.673552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.673535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.706116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.706087 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.775962 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblth\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776035 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776065 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776125 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776204 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776384 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776512 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.776605 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.781880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.777119 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.810911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.809113 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.810911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.809379 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.810911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.809602 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.810911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.810153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.845924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.843950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblth\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.849888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.847472 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.873933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.852366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.873933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.853366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\") pod \"e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:49.880886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.876948 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:49.949647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:49.949606 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:50.085754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.085717 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:50.291789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.291759 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:10:50.291789 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:50.291788 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0bab2e_83ac_421d_a89b_1491900d0977.slice/crio-d328057a4e0707c9577c4fc04d9e42fb80016473eb6e44e8072145ed82853083 WatchSource:0}: Error finding container d328057a4e0707c9577c4fc04d9e42fb80016473eb6e44e8072145ed82853083: Status 404 returned error can't find the container with id d328057a4e0707c9577c4fc04d9e42fb80016473eb6e44e8072145ed82853083 Oct 30 01:10:50.295616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.295579 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:50.295755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.295657 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:50.295755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.295701 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:50.885080 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:50.885043 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.105190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.105160 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:51.118479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.118454 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:51.139035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.138964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.212187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.212158 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerStarted","Data":"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680"} Oct 30 01:10:51.212331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.212198 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerStarted","Data":"d328057a4e0707c9577c4fc04d9e42fb80016473eb6e44e8072145ed82853083"} Oct 30 01:10:51.223369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.223342 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:10:51.258313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.258275 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podStartSLOduration=2.258263405 podStartE2EDuration="2.258263405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:51.257027291 +0000 UTC m=+2303.497262408" watchObservedRunningTime="2026-10-30 01:10:51.258263405 +0000 UTC m=+2303.498498559" Oct 30 01:10:51.950758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.950727 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:51.956756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.956734 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:51.989334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:51.987114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:52.221524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.221436 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:52.222321 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.222297 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:10:52.508965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.508883 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:52.593794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.593766 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:10:52.598108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.598085 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.616176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.616152 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a3619c\"" Oct 30 01:10:52.616449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.616430 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:10:52.654237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.654213 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:10:52.657542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.657522 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.666818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.666792 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:52.676062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.676039 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a3619c-dockercfg-njhmr\"" Oct 30 01:10:52.677164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.677146 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:10:52.768937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.755905 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.768937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.756028 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvkc\" (UniqueName: \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.768937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.756156 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.768937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.756207 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7pcj\" (UniqueName: \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.875888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.865393 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.875888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.865516 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvkc\" (UniqueName: \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.875888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.865652 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.875888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.865698 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7pcj\" (UniqueName: \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.876275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.875988 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.912272 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.912229 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.932363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.932328 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7pcj\" (UniqueName: \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\") pod \"payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:52.934476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.934443 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvkc\" (UniqueName: \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\") pod \"payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:52.972331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:52.972306 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:53.216007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.215979 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:53.277038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.276245 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:10:53.523488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.523462 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:10:53.524592 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:53.524567 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655a8378_d816_40a9_832b_25b1b6b211a9.slice/crio-12ef92ffe2dcd3cc9ceb8f86007c03d139df3dfaf1c7ceffd50e6b5690c8ebec WatchSource:0}: Error finding container 12ef92ffe2dcd3cc9ceb8f86007c03d139df3dfaf1c7ceffd50e6b5690c8ebec: Status 404 returned error can't find the container with id 12ef92ffe2dcd3cc9ceb8f86007c03d139df3dfaf1c7ceffd50e6b5690c8ebec Oct 30 01:10:53.759846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.759815 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:53.771197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.771173 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:53.772892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.772840 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:53.982694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:53.973377 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.242099 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.242006 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" event={"ID":"655a8378-d816-40a9-832b-25b1b6b211a9","Type":"ContainerStarted","Data":"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e"} Oct 30 01:10:54.242099 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.242074 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:10:54.242099 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.242099 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" event={"ID":"655a8378-d816-40a9-832b-25b1b6b211a9","Type":"ContainerStarted","Data":"12ef92ffe2dcd3cc9ceb8f86007c03d139df3dfaf1c7ceffd50e6b5690c8ebec"} Oct 30 01:10:54.243807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.243776 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" event={"ID":"4236541a-c30a-4c70-8220-57f280c7389d","Type":"ContainerStarted","Data":"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af"} Oct 30 01:10:54.244250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.244227 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:10:54.244347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.244271 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" event={"ID":"4236541a-c30a-4c70-8220-57f280c7389d","Type":"ContainerStarted","Data":"384152c2a570930a93af780ce52a5b28ca99d20a1bede5f62361f63c0322d333"} Oct 30 01:10:54.255125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.255099 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:10:54.278071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.278044 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.295313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.295273 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" podStartSLOduration=2.295261409 podStartE2EDuration="2.295261409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:54.280826137 +0000 UTC m=+2306.521061244" watchObservedRunningTime="2026-10-30 01:10:54.295261409 +0000 UTC m=+2306.535496522" Oct 30 01:10:54.324741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.324695 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" podStartSLOduration=2.324682753 podStartE2EDuration="2.324682753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:54.323830997 +0000 UTC m=+2306.564066111" watchObservedRunningTime="2026-10-30 01:10:54.324682753 +0000 UTC m=+2306.564917867" Oct 30 01:10:54.680056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.680017 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:10:54.685101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.685074 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.711338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.711311 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:10:54.711587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.711561 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d8b2c6-openshift-default-dockercfg-ltz9d\"" Oct 30 01:10:54.788972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.788936 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.851798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.851772 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.851983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.851830 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjnz\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.851983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.851946 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.851985 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.852195 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.852320 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.852355 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.852479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.852567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.852526 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957283 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957402 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957452 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957505 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957534 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957622 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.957940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.957704 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjnz\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.965887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.962804 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.965887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.963120 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.967770 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:54.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.967938 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.968599 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.968851 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.971886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.969420 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:54.994887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:54.990645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:55.015885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.015340 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:55.060884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.047844 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:55.060884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.052746 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjnz\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\") pod \"e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:55.306276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.306200 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:55.589562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.588169 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:55.625907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.625880 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:10:55.627326 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:55.627301 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34964578_4b4e_4361_aa9b_cc61ce9f9110.slice/crio-d1ae4c3a49b43fbeb6ec108a4c95c9d6a6b7678011c0a2d6b3a8b34c63629d8b WatchSource:0}: Error finding container d1ae4c3a49b43fbeb6ec108a4c95c9d6a6b7678011c0a2d6b3a8b34c63629d8b: Status 404 returned error can't find the container with id d1ae4c3a49b43fbeb6ec108a4c95c9d6a6b7678011c0a2d6b3a8b34c63629d8b Oct 30 01:10:55.630234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.630199 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:55.630315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.630267 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:55.630315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.630298 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:55.669621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.669597 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:10:55.674055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.674033 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.688419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.688389 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a7ca1f\"" Oct 30 01:10:55.690757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.690731 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:10:55.723461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.723439 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:55.728939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.728917 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:10:55.732470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.732453 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:55.753612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.753588 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a7ca1f-dockercfg-nw8wn\"" Oct 30 01:10:55.755768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.755730 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:10:55.784439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.784408 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.784574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.784463 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcxt\" (UniqueName: \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.922893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.906188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlt4r\" (UniqueName: \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:55.922893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.906324 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:55.922893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.906443 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.922893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.906482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcxt\" (UniqueName: \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.922893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.913746 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:55.931017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.930820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:55.953884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:55.951056 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcxt\" (UniqueName: \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\") pod \"payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:56.015304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.015249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlt4r\" (UniqueName: \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:56.015806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.015407 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:56.016649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.016525 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:56.016791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.016697 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:56.077057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.076934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlt4r\" (UniqueName: \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:56.268121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.268038 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerStarted","Data":"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15"} Oct 30 01:10:56.268327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.268137 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerStarted","Data":"d1ae4c3a49b43fbeb6ec108a4c95c9d6a6b7678011c0a2d6b3a8b34c63629d8b"} Oct 30 01:10:56.306502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.306456 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:56.308094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.308068 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:10:56.308228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.308111 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:10:56.315755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.315709 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podStartSLOduration=2.315695924 podStartE2EDuration="2.315695924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:56.311419176 +0000 UTC m=+2308.551654291" watchObservedRunningTime="2026-10-30 01:10:56.315695924 +0000 UTC m=+2308.555931034" Oct 30 01:10:56.348978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.348955 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:56.383411 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:56.383379 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2987fc_83e2_49ee_946b_10acfd1668aa.slice/crio-c5f8d37ac9b9649d7f0f6f50657c6f7abcaba21b32b812899d3b8108b86311fb WatchSource:0}: Error finding container c5f8d37ac9b9649d7f0f6f50657c6f7abcaba21b32b812899d3b8108b86311fb: Status 404 returned error can't find the container with id c5f8d37ac9b9649d7f0f6f50657c6f7abcaba21b32b812899d3b8108b86311fb Oct 30 01:10:56.383734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.383681 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:10:56.660835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:56.660802 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:10:56.661493 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:56.661472 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a1f548_92d1_42e7_9838_53021764e613.slice/crio-fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95 WatchSource:0}: Error finding container fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95: Status 404 returned error can't find the container with id fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95 Oct 30 01:10:57.278894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.278821 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" event={"ID":"fd2987fc-83e2-49ee-946b-10acfd1668aa","Type":"ContainerStarted","Data":"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85"} Oct 30 01:10:57.279366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.278917 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" event={"ID":"fd2987fc-83e2-49ee-946b-10acfd1668aa","Type":"ContainerStarted","Data":"c5f8d37ac9b9649d7f0f6f50657c6f7abcaba21b32b812899d3b8108b86311fb"} Oct 30 01:10:57.279366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.278968 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:10:57.281178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.281149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" event={"ID":"79a1f548-92d1-42e7-9838-53021764e613","Type":"ContainerStarted","Data":"fdea9eb7619f679b61a9347e1f4a8e6f795f62766bc31b1703e28304bedbca13"} Oct 30 01:10:57.281331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.281204 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" event={"ID":"79a1f548-92d1-42e7-9838-53021764e613","Type":"ContainerStarted","Data":"fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95"} Oct 30 01:10:57.281434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.281413 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:10:57.312217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.312191 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:57.321061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.321022 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" podStartSLOduration=2.321009538 podStartE2EDuration="2.321009538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:57.318371101 +0000 UTC m=+2309.558606218" watchObservedRunningTime="2026-10-30 01:10:57.321009538 +0000 UTC m=+2309.561244650" Oct 30 01:10:57.359289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.359240 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" podStartSLOduration=2.359227717 podStartE2EDuration="2.359227717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:57.356395109 +0000 UTC m=+2309.596630222" watchObservedRunningTime="2026-10-30 01:10:57.359227717 +0000 UTC m=+2309.599462829" Oct 30 01:10:57.833483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:57.833446 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:58.010029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.009995 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:10:58.013820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.013802 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.031533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.031511 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-73435a-openshift-default-dockercfg-whl5c\"" Oct 30 01:10:58.050338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050298 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050348 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050400 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050464 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050539 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlvl\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050637 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050747 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.050802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050780 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.051006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.050810 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.051543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.051521 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:10:58.151807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.151762 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.151985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.151911 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.151985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.151954 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152103 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.151982 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152117 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152144 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152195 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152270 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152354 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlvl\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152560 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152605 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152815 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.152924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.152858 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.153352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.153330 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.154939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.154915 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.155048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.155033 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.174586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.174565 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.175199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.175177 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlvl\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.290687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.290607 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:58.291515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.291491 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:10:58.328974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.328949 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:58.724010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.723980 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:10:58.724284 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:10:58.724264 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41579ccd_347f_402a_9fa5_19dc639875e9.slice/crio-e16de1699710712e6e5f15a10d810c1d68ea53039378b1e7f8104da2ae892d17 WatchSource:0}: Error finding container e16de1699710712e6e5f15a10d810c1d68ea53039378b1e7f8104da2ae892d17: Status 404 returned error can't find the container with id e16de1699710712e6e5f15a10d810c1d68ea53039378b1e7f8104da2ae892d17 Oct 30 01:10:58.727217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.727187 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:58.727300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.727250 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:58.727300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.727278 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:10:58.794189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.794159 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:58.840078 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:10:58.840035 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:10:58.944326 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:58.944289 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:59.322636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.322598 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" event={"ID":"41579ccd-347f-402a-9fa5-19dc639875e9","Type":"ContainerStarted","Data":"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b"} Oct 30 01:10:59.323192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.322670 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" event={"ID":"41579ccd-347f-402a-9fa5-19dc639875e9","Type":"ContainerStarted","Data":"e16de1699710712e6e5f15a10d810c1d68ea53039378b1e7f8104da2ae892d17"} Oct 30 01:10:59.336027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.334336 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:10:59.336950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.336923 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:10:59.337060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.336979 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:10:59.397899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.396407 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podStartSLOduration=2.396386929 podStartE2EDuration="2.396386929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:59.376836031 +0000 UTC m=+2311.617071157" watchObservedRunningTime="2026-10-30 01:10:59.396386929 +0000 UTC m=+2311.636622043" Oct 30 01:10:59.919727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:10:59.919691 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.329339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.329260 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:11:00.329339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.329318 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:11:00.480623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.480580 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.623554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.623521 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:11:00.627239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.627219 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.646479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.646452 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:11:00.650456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.650415 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.653401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.653382 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d8b2c6\"" Oct 30 01:11:00.653533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.653453 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d8b2c6-dockercfg-vf9vf\"" Oct 30 01:11:00.763886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.755796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.763886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.756134 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ck4d\" (UniqueName: \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.824556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.824521 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:11:00.921936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.918971 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ck4d\" (UniqueName: \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.921936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.919256 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.921936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.920369 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:00.981166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.981138 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:00.982180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:00.982153 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:11:01.016372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.016343 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ck4d\" (UniqueName: \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:01.123340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.123309 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4wm\" (UniqueName: \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.123530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.123512 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.225034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.224939 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.225222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.225174 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4wm\" (UniqueName: \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.225685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.225661 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.241244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.241218 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:01.247915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.247854 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4wm\" (UniqueName: \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\") pod \"payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.330802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.330762 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:11:01.331157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.330826 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:11:01.333554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.333533 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:01.579694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.579665 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:11:01.581059 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:11:01.581033 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb25f59_b949_474d_a793_8e9714cee19e.slice/crio-55853f984130ee50f9bc14e76d965b92888c8b6c0f6344991679231f87c0d134 WatchSource:0}: Error finding container 55853f984130ee50f9bc14e76d965b92888c8b6c0f6344991679231f87c0d134: Status 404 returned error can't find the container with id 55853f984130ee50f9bc14e76d965b92888c8b6c0f6344991679231f87c0d134 Oct 30 01:11:01.656473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.656443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:01.730377 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:11:01.730345 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbd727b_d4ab_4009_bfd1_ca604e1c1971.slice/crio-f24622e8106dc9a7b541b4105c8319adde9cf1499f11a694d363f18290bbcc79 WatchSource:0}: Error finding container f24622e8106dc9a7b541b4105c8319adde9cf1499f11a694d363f18290bbcc79: Status 404 returned error can't find the container with id f24622e8106dc9a7b541b4105c8319adde9cf1499f11a694d363f18290bbcc79 Oct 30 01:11:01.732529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:01.732341 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:11:02.329737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.329700 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:11:02.329938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.329756 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:11:02.354673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.354631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" event={"ID":"3eb25f59-b949-474d-a793-8e9714cee19e","Type":"ContainerStarted","Data":"fa3321b4bcb9bafd76fa375f010d2041bd61f9c482ab168d74bbab662e7773f9"} Oct 30 01:11:02.355114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.354728 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" event={"ID":"3eb25f59-b949-474d-a793-8e9714cee19e","Type":"ContainerStarted","Data":"55853f984130ee50f9bc14e76d965b92888c8b6c0f6344991679231f87c0d134"} Oct 30 01:11:02.355114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.354757 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:02.356520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.356493 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" event={"ID":"0dbd727b-d4ab-4009-bfd1-ca604e1c1971","Type":"ContainerStarted","Data":"09fbe2f5bbef25f19bbac04c865344f8f11f46f1d54beb3fe51f46de93f49e95"} Oct 30 01:11:02.356602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.356546 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:02.356602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.356564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" event={"ID":"0dbd727b-d4ab-4009-bfd1-ca604e1c1971","Type":"ContainerStarted","Data":"f24622e8106dc9a7b541b4105c8319adde9cf1499f11a694d363f18290bbcc79"} Oct 30 01:11:02.409885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.409107 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" podStartSLOduration=2.409090943 podStartE2EDuration="2.409090943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:02.399218719 +0000 UTC m=+2314.639453833" watchObservedRunningTime="2026-10-30 01:11:02.409090943 +0000 UTC m=+2314.649326056" Oct 30 01:11:02.442180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:02.442136 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" podStartSLOduration=2.442123842 podStartE2EDuration="2.442123842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:02.437627168 +0000 UTC m=+2314.677862283" watchObservedRunningTime="2026-10-30 01:11:02.442123842 +0000 UTC m=+2314.682358955" Oct 30 01:11:03.330115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:03.330080 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:11:03.330294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:03.330131 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:11:03.377253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:03.377223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.329459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:04.329431 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 01:11:04.329617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:04.329481 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 01:11:05.181745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:05.181714 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:05.261557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:05.261526 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:11:05.464566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:05.464478 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:11:05.464929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:05.464906 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:11:05.466372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:05.466347 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:11:06.271297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:06.271271 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:11:06.856810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:06.856762 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.294128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:08.294099 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:11:08.294521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:08.294171 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:11:08.475049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:08.475019 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:10.278415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:10.278373 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:11.969991 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:11.969951 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:13.267899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.267858 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:13.398070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.398039 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:11:13.410232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.410196 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:11:13.451560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.451524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:13.818793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.818757 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:11:13.886606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.886576 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:13.933601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.922238 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f667-dockercfg-pk7h5\"" Oct 30 01:11:13.933601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.922486 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:13.933601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.922574 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f667\"" Oct 30 01:11:13.933601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.922783 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnls\" (UniqueName: \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:13.949150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:13.949023 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:11:14.027731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.027688 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:14.027983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.027845 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnls\" (UniqueName: \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:14.028762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.028736 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:14.050603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.050577 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnls\" (UniqueName: \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\") pod \"payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:14.241458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.241414 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:14.441995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.441956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:14.622070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:14.622040 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:11:14.667020 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:11:14.666985 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a67a99_4b4f_4ca7_8b0c_cda119aded82.slice/crio-e87b59322f8604c20944d13e686b9effc0eec26d418160a133d00a571fa55edc WatchSource:0}: Error finding container e87b59322f8604c20944d13e686b9effc0eec26d418160a133d00a571fa55edc: Status 404 returned error can't find the container with id e87b59322f8604c20944d13e686b9effc0eec26d418160a133d00a571fa55edc Oct 30 01:11:15.512847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:15.512813 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" event={"ID":"61a67a99-4b4f-4ca7-8b0c-cda119aded82","Type":"ContainerStarted","Data":"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1"} Oct 30 01:11:15.513438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:15.513413 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" event={"ID":"61a67a99-4b4f-4ca7-8b0c-cda119aded82","Type":"ContainerStarted","Data":"e87b59322f8604c20944d13e686b9effc0eec26d418160a133d00a571fa55edc"} Oct 30 01:11:15.514599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:15.514578 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:17.002028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:17.001997 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:18.586609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:18.586574 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.371721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:20.371685 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:22.169415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:22.169378 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:22.311471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:22.311440 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:23.822654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:23.822621 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:25.633022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:25.632654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:27.374537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:27.374495 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:27.536534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:27.536502 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:11:27.580399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:27.580353 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" podStartSLOduration=14.580342056 podStartE2EDuration="14.580342056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:15.561109363 +0000 UTC m=+2327.801344478" watchObservedRunningTime="2026-10-30 01:11:27.580342056 +0000 UTC m=+2339.820577168" Oct 30 01:11:28.836351 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:11:28.836312 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:11:29.105610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:29.105420 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:29.427421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:29.427343 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:29.881123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:29.881094 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:30.034775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:30.034739 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:30.887759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:30.887722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:32.614670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:32.614609 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:32.755911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:32.755831 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:33.252642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:33.252581 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:33.483959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:33.483920 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:34.733665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:34.733628 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:34.958162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:34.958128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.586011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.585970 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.660697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.660669 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:11:35.666829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.666808 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.683132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.683104 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:11:35.688538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.688515 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6d44c2-dockercfg-9k422\"" Oct 30 01:11:35.688538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.688521 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6d44c2\"" Oct 30 01:11:35.712965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.712934 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:11:35.718959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.718937 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:35.736643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.736620 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:11:35.743819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.743797 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.765325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.765297 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8wh\" (UniqueName: \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.765550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.765528 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.901925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.886923 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:35.901925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.887008 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5s6\" (UniqueName: \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:35.901925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.887163 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8wh\" (UniqueName: \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.901925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.887236 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.902741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.902663 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:35.989695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.989480 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:35.990108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.990075 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5s6\" (UniqueName: \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:35.992340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:35.992289 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:36.003641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.003615 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8wh\" (UniqueName: \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\") pod \"payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:36.016179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.016154 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5s6\" (UniqueName: \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:36.039036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.039011 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:36.282277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.282191 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:36.358574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.357995 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:11:36.508117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.508089 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:36.588662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.588633 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:11:36.590353 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:11:36.590324 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1ead2f_a44d_43f9_beed_7f16319447bb.slice/crio-a87357c49909377772a83e1e2525f5a30a8a17443136d6256a2cc8408adf63c8 WatchSource:0}: Error finding container a87357c49909377772a83e1e2525f5a30a8a17443136d6256a2cc8408adf63c8: Status 404 returned error can't find the container with id a87357c49909377772a83e1e2525f5a30a8a17443136d6256a2cc8408adf63c8 Oct 30 01:11:36.699472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.699432 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" event={"ID":"be1ead2f-a44d-43f9-beed-7f16319447bb","Type":"ContainerStarted","Data":"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1"} Oct 30 01:11:36.699944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.699498 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" event={"ID":"be1ead2f-a44d-43f9-beed-7f16319447bb","Type":"ContainerStarted","Data":"a87357c49909377772a83e1e2525f5a30a8a17443136d6256a2cc8408adf63c8"} Oct 30 01:11:36.699944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.699527 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:36.701395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.701369 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" event={"ID":"de414aea-a10a-4331-a392-e34b42996819","Type":"ContainerStarted","Data":"8b1928ab01f7c1697e17f0bc310cd68514fe84251bea82fcd278ae07d209cc26"} Oct 30 01:11:36.701517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.701417 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" event={"ID":"de414aea-a10a-4331-a392-e34b42996819","Type":"ContainerStarted","Data":"d0eae1ffb4639c975b0bf08a89fd04c6f7b5d2f0f0f860f753c91e261bf723d5"} Oct 30 01:11:36.701517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.701453 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:36.740117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.740070 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" podStartSLOduration=1.7400580319999999 podStartE2EDuration="1.740058032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:36.737824348 +0000 UTC m=+2348.978059463" watchObservedRunningTime="2026-10-30 01:11:36.740058032 +0000 UTC m=+2348.980293155" Oct 30 01:11:36.777759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:36.777697 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" podStartSLOduration=1.777684545 podStartE2EDuration="1.777684545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:36.77611421 +0000 UTC m=+2349.016349335" watchObservedRunningTime="2026-10-30 01:11:36.777684545 +0000 UTC m=+2349.017919658" Oct 30 01:11:38.447203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:38.447169 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:38.586209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:38.586171 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:39.264855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:39.264721 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.080292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:40.080254 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.214604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:40.214572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:40.530522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:40.529270 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:42.374445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:42.374411 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:44.237938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:44.237898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:44.385896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:44.385835 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:46.176901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:46.176852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:47.712227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:47.712195 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:11:47.712616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:47.712423 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:11:47.887687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:47.887650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:48.121852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:48.121823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:49.662051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:49.662004 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:51.431729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:51.431681 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:53.047350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:53.047311 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.808682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:54.808642 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:56.494539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:56.494502 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:58.103959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:58.103913 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:58.836158 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:11:58.836126 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:11:59.923776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:11:59.923696 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:02.319489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:02.319455 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:04.915434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:04.915394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:06.682519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:06.682485 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:08.321890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:08.319371 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:09.612985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:09.612934 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:09.926592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:09.926503 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:11.697455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:11.697380 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:13.335664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:13.335626 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:13.593601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:13.593509 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:15.193956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:15.139224 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:16.170602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.170572 5938 generic.go:356] "Generic (PLEG): container finished" podID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerID="c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b" exitCode=0 Oct 30 01:12:16.170788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.170652 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerDied","Data":"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b"} Oct 30 01:12:16.171213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.171192 5938 scope.go:122] "RemoveContainer" containerID="c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b" Oct 30 01:12:16.173149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.173116 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:16.173269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.173204 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:16.173269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.173256 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:16.976658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:16.976617 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:17.183049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:17.183014 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerStarted","Data":"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6"} Oct 30 01:12:17.897886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:17.896059 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:12:17.897886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:17.897740 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 01:12:17.897886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:17.897804 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 01:12:18.225944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.225844 5938 generic.go:356] "Generic (PLEG): container finished" podID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerID="28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680" exitCode=0 Oct 30 01:12:18.226368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.225918 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerDied","Data":"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680"} Oct 30 01:12:18.226437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.226411 5938 scope.go:122] "RemoveContainer" containerID="28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680" Oct 30 01:12:18.228432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.228404 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:18.228554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.228491 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:18.228554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.228532 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:12:18.616391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.616354 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.910136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.910029 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 01:12:18.910136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:18.910112 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 01:12:19.199940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.196969 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:12:19.254349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.254314 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerStarted","Data":"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e"} Oct 30 01:12:19.555690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.555579 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:19.705385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.705335 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:19.897239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.897203 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 01:12:19.897540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.897506 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 01:12:19.950314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.950276 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:12:19.951790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.951763 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 01:12:19.951958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:19.951813 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 01:12:20.343517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.343429 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:20.504527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.504493 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:20.650253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.650217 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:20.896377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.896339 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 01:12:20.896722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.896692 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 01:12:20.950985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.950907 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 01:12:20.950985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:20.950964 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 01:12:21.896888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:21.896186 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body= Oct 30 01:12:21.896888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:21.896258 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" Oct 30 01:12:21.950771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:21.950736 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 01:12:21.950958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:21.950782 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 01:12:22.160095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.160014 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:22.230999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.230962 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:12:22.913277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.913249 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:12:22.914345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.914320 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:12:22.950177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.950149 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 01:12:22.950300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:22.950204 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 01:12:23.876766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:23.876725 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:23.951353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:23.951306 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 30 01:12:23.951890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:23.951392 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 30 01:12:25.034763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:25.034722 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:12:25.036834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:25.036806 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:12:25.724336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:25.724304 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:27.505250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:27.505212 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:28.365181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:28.365138 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:28.843921 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:12:28.843810 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:12:29.139448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:29.139410 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:30.567416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:30.567389 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:12:30.593058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:30.593034 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:12:31.066350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.066323 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:12:31.091605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.091580 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:31.092461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.092441 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:12:31.241526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.241494 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:31.381457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.381414 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:31.530829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:31.530799 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:33.093313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:33.093261 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:34.034231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:34.034201 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:34.802547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:34.802512 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.093346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.093304 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" podUID="79a1f548-92d1-42e7-9838-53021764e613" containerName="payload-pre-processing" containerID="cri-o://fdea9eb7619f679b61a9347e1f4a8e6f795f62766bc31b1703e28304bedbca13" gracePeriod=25 Oct 30 01:12:36.295494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.295413 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:12:36.300604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.300579 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:12:36.388999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.388961 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" podUID="fd2987fc-83e2-49ee-946b-10acfd1668aa" containerName="payload-processing" containerID="cri-o://2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85" gracePeriod=25 Oct 30 01:12:36.412020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.411986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.429050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.429027 5938 generic.go:356] "Generic (PLEG): container finished" podID="79a1f548-92d1-42e7-9838-53021764e613" containerID="fdea9eb7619f679b61a9347e1f4a8e6f795f62766bc31b1703e28304bedbca13" exitCode=0 Oct 30 01:12:36.429150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.429098 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" event={"ID":"79a1f548-92d1-42e7-9838-53021764e613","Type":"ContainerDied","Data":"fdea9eb7619f679b61a9347e1f4a8e6f795f62766bc31b1703e28304bedbca13"} Oct 30 01:12:36.429197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.429147 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" event={"ID":"79a1f548-92d1-42e7-9838-53021764e613","Type":"ContainerDied","Data":"fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95"} Oct 30 01:12:36.429234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.429197 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe126d4d388caad77e22814364a55d0e9b1228ca06cf691ce00683bcfd0c7d95" Oct 30 01:12:36.507969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.507927 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:12:36.555469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.555433 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.607024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.606768 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\") pod \"79a1f548-92d1-42e7-9838-53021764e613\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " Oct 30 01:12:36.607242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.607044 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\" (UniqueName: \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\") pod \"79a1f548-92d1-42e7-9838-53021764e613\" (UID: \"79a1f548-92d1-42e7-9838-53021764e613\") " Oct 30 01:12:36.612004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.611977 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r" pod "79a1f548-92d1-42e7-9838-53021764e613" (UID: "79a1f548-92d1-42e7-9838-53021764e613"). InnerVolumeSpecName "kube-api-access-qlt4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:36.613157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.613124 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume" pod "79a1f548-92d1-42e7-9838-53021764e613" (UID: "79a1f548-92d1-42e7-9838-53021764e613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:36.705416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.705382 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.710125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.710102 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a1f548-92d1-42e7-9838-53021764e613-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:36.710275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.710266 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlt4r\" (UniqueName: \"kubernetes.io/projected/79a1f548-92d1-42e7-9838-53021764e613-kube-api-access-qlt4r\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:36.724629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.724608 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:12:36.811502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.811465 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\") pod \"fd2987fc-83e2-49ee-946b-10acfd1668aa\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " Oct 30 01:12:36.811767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.811611 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\" (UniqueName: \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\") pod \"fd2987fc-83e2-49ee-946b-10acfd1668aa\" (UID: \"fd2987fc-83e2-49ee-946b-10acfd1668aa\") " Oct 30 01:12:36.811918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.811892 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume" pod "fd2987fc-83e2-49ee-946b-10acfd1668aa" (UID: "fd2987fc-83e2-49ee-946b-10acfd1668aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:36.812387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.812362 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd2987fc-83e2-49ee-946b-10acfd1668aa-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:36.813858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.813827 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt" pod "fd2987fc-83e2-49ee-946b-10acfd1668aa" (UID: "fd2987fc-83e2-49ee-946b-10acfd1668aa"). InnerVolumeSpecName "kube-api-access-kqcxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:36.919947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:36.919307 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcxt\" (UniqueName: \"kubernetes.io/projected/fd2987fc-83e2-49ee-946b-10acfd1668aa-kube-api-access-kqcxt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:37.440043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440008 5938 generic.go:356] "Generic (PLEG): container finished" podID="fd2987fc-83e2-49ee-946b-10acfd1668aa" containerID="2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85" exitCode=0 Oct 30 01:12:37.440420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440056 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" Oct 30 01:12:37.440420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440092 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" event={"ID":"fd2987fc-83e2-49ee-946b-10acfd1668aa","Type":"ContainerDied","Data":"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85"} Oct 30 01:12:37.440420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj" event={"ID":"fd2987fc-83e2-49ee-946b-10acfd1668aa","Type":"ContainerDied","Data":"c5f8d37ac9b9649d7f0f6f50657c6f7abcaba21b32b812899d3b8108b86311fb"} Oct 30 01:12:37.440420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440169 5938 scope.go:122] "RemoveContainer" containerID="2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85" Oct 30 01:12:37.440601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.440448 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h" Oct 30 01:12:37.454057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.454009 5938 scope.go:122] "RemoveContainer" containerID="2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85" Oct 30 01:12:37.454324 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:12:37.454305 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85\": container with ID starting with 2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85 not found: ID does not exist" containerID="2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85" Oct 30 01:12:37.454390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.454332 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85"} err="failed to get container status \"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85\": rpc error: code = NotFound desc = could not find container \"2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85\": container with ID starting with 2fa2e9938b53925b99c83b1e262bdeca046e13c695bd5b419b83709c87bb3c85 not found: ID does not exist" Oct 30 01:12:37.491467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.491442 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:12:37.502783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.502758 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-788dfbb58c-dt82h"] Oct 30 01:12:37.527848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.527827 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:12:37.531441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.531422 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-8558789598-6jrqj"] Oct 30 01:12:37.879665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.879628 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a1f548-92d1-42e7-9838-53021764e613" path="/var/lib/kubelet/pods/79a1f548-92d1-42e7-9838-53021764e613/volumes" Oct 30 01:12:37.880179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:37.880156 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2987fc-83e2-49ee-946b-10acfd1668aa" path="/var/lib/kubelet/pods/fd2987fc-83e2-49ee-946b-10acfd1668aa/volumes" Oct 30 01:12:38.402527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:38.402483 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:40.208125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:40.208091 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:40.354609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:40.354577 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:41.296192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.296155 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" podUID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" containerName="payload-processing" containerID="cri-o://09fbe2f5bbef25f19bbac04c865344f8f11f46f1d54beb3fe51f46de93f49e95" gracePeriod=25 Oct 30 01:12:41.301154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.301122 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" podUID="3eb25f59-b949-474d-a793-8e9714cee19e" containerName="payload-pre-processing" containerID="cri-o://fa3321b4bcb9bafd76fa375f010d2041bd61f9c482ab168d74bbab662e7773f9" gracePeriod=25 Oct 30 01:12:41.477572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.477542 5938 generic.go:356] "Generic (PLEG): container finished" podID="3eb25f59-b949-474d-a793-8e9714cee19e" containerID="fa3321b4bcb9bafd76fa375f010d2041bd61f9c482ab168d74bbab662e7773f9" exitCode=0 Oct 30 01:12:41.478825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.478796 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" event={"ID":"3eb25f59-b949-474d-a793-8e9714cee19e","Type":"ContainerDied","Data":"fa3321b4bcb9bafd76fa375f010d2041bd61f9c482ab168d74bbab662e7773f9"} Oct 30 01:12:41.480800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.480772 5938 generic.go:356] "Generic (PLEG): container finished" podID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" containerID="09fbe2f5bbef25f19bbac04c865344f8f11f46f1d54beb3fe51f46de93f49e95" exitCode=0 Oct 30 01:12:41.480947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.480816 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" event={"ID":"0dbd727b-d4ab-4009-bfd1-ca604e1c1971","Type":"ContainerDied","Data":"09fbe2f5bbef25f19bbac04c865344f8f11f46f1d54beb3fe51f46de93f49e95"} Oct 30 01:12:41.615907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.615880 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:12:41.653940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.653918 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:12:41.681034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.681008 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\") pod \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " Oct 30 01:12:41.681254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.681234 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\" (UniqueName: \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\") pod \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\" (UID: \"0dbd727b-d4ab-4009-bfd1-ca604e1c1971\") " Oct 30 01:12:41.681393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.681368 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume" pod "0dbd727b-d4ab-4009-bfd1-ca604e1c1971" (UID: "0dbd727b-d4ab-4009-bfd1-ca604e1c1971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:41.682029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.682005 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:41.683413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.683391 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm" pod "0dbd727b-d4ab-4009-bfd1-ca604e1c1971" (UID: "0dbd727b-d4ab-4009-bfd1-ca604e1c1971"). InnerVolumeSpecName "kube-api-access-hf4wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:41.783327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.783305 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\") pod \"3eb25f59-b949-474d-a793-8e9714cee19e\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " Oct 30 01:12:41.783458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.783360 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\" (UniqueName: \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\") pod \"3eb25f59-b949-474d-a793-8e9714cee19e\" (UID: \"3eb25f59-b949-474d-a793-8e9714cee19e\") " Oct 30 01:12:41.783685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.783665 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume" pod "3eb25f59-b949-474d-a793-8e9714cee19e" (UID: "3eb25f59-b949-474d-a793-8e9714cee19e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:41.783754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.783729 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4wm\" (UniqueName: \"kubernetes.io/projected/0dbd727b-d4ab-4009-bfd1-ca604e1c1971-kube-api-access-hf4wm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:41.785626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.785606 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d" pod "3eb25f59-b949-474d-a793-8e9714cee19e" (UID: "3eb25f59-b949-474d-a793-8e9714cee19e"). InnerVolumeSpecName "kube-api-access-6ck4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:41.884809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.884782 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb25f59-b949-474d-a793-8e9714cee19e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:41.884958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.884816 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ck4d\" (UniqueName: \"kubernetes.io/projected/3eb25f59-b949-474d-a793-8e9714cee19e-kube-api-access-6ck4d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:12:41.994303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:41.994280 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:42.490252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.490225 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" Oct 30 01:12:42.490697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.490270 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn" event={"ID":"3eb25f59-b949-474d-a793-8e9714cee19e","Type":"ContainerDied","Data":"55853f984130ee50f9bc14e76d965b92888c8b6c0f6344991679231f87c0d134"} Oct 30 01:12:42.490697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.490338 5938 scope.go:122] "RemoveContainer" containerID="fa3321b4bcb9bafd76fa375f010d2041bd61f9c482ab168d74bbab662e7773f9" Oct 30 01:12:42.491959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.491938 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" Oct 30 01:12:42.492098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.491963 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756" event={"ID":"0dbd727b-d4ab-4009-bfd1-ca604e1c1971","Type":"ContainerDied","Data":"f24622e8106dc9a7b541b4105c8319adde9cf1499f11a694d363f18290bbcc79"} Oct 30 01:12:42.501606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.501588 5938 scope.go:122] "RemoveContainer" containerID="09fbe2f5bbef25f19bbac04c865344f8f11f46f1d54beb3fe51f46de93f49e95" Oct 30 01:12:42.544922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.544899 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:12:42.556147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.556124 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-6fd666c55b-4r756"] Oct 30 01:12:42.583577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.583512 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:12:42.589452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:42.589433 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bf667b7f5-rrwwn"] Oct 30 01:12:43.697961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:43.697930 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:43.879910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:43.879878 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" path="/var/lib/kubelet/pods/0dbd727b-d4ab-4009-bfd1-ca604e1c1971/volumes" Oct 30 01:12:43.880302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:43.880287 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb25f59-b949-474d-a793-8e9714cee19e" path="/var/lib/kubelet/pods/3eb25f59-b949-474d-a793-8e9714cee19e/volumes" Oct 30 01:12:45.130856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:45.130818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:45.591913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:45.591816 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:47.305259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:47.305221 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:48.982296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:48.982265 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:49.158748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:49.158714 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:51.598421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:51.598383 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:53.940814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:53.940778 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:55.709972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:55.709934 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:57.503887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:57.500368 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:58.847167 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:12:58.847130 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:12:59.169857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:59.169811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:59.333028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:59.332984 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:12:59.333225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:12:59.333042 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:13:00.803819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:00.803779 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:02.678520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.678485 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:02.709852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.709814 5938 generic.go:356] "Generic (PLEG): container finished" podID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerID="0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15" exitCode=0 Oct 30 01:13:02.710174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.710041 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerDied","Data":"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15"} Oct 30 01:13:02.710769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.710746 5938 scope.go:122] "RemoveContainer" containerID="0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15" Oct 30 01:13:02.713363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.713326 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:02.713494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.713410 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:02.713494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:02.713448 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:03.721729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:03.721686 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerStarted","Data":"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87"} Oct 30 01:13:04.308888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:04.308469 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:13:04.308888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:04.308628 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:13:04.308888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:04.308670 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:13:04.473117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:04.473080 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:05.312279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:05.312231 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:13:05.312789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:05.312310 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:13:06.210844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:06.210814 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:06.309608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:06.309563 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:13:06.309820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:06.309649 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:13:07.309190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:07.309158 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:13:07.309931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:07.309899 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:13:07.910142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:07.910109 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:08.309311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:08.309228 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 01:13:08.309898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:08.309848 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 01:13:09.311015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:09.310939 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:13:09.311462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:09.311444 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:13:09.312071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:09.312052 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:13:09.582201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:09.582126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:11.462483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:11.462451 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:13.302739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:13.302661 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:14.854880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.854835 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:13:14.856096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856068 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" containerName="payload-processing" Oct 30 01:13:14.856096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856093 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" containerName="payload-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856135 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3eb25f59-b949-474d-a793-8e9714cee19e" containerName="payload-pre-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856145 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb25f59-b949-474d-a793-8e9714cee19e" containerName="payload-pre-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856194 5938 cpu_manager.go:414] "containerMap: removing container" podUID="79a1f548-92d1-42e7-9838-53021764e613" containerName="payload-pre-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856203 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a1f548-92d1-42e7-9838-53021764e613" containerName="payload-pre-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856215 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fd2987fc-83e2-49ee-946b-10acfd1668aa" containerName="payload-processing" Oct 30 01:13:14.856304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856223 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2987fc-83e2-49ee-946b-10acfd1668aa" containerName="payload-processing" Oct 30 01:13:14.856629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856381 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbd727b-d4ab-4009-bfd1-ca604e1c1971" containerName="payload-processing" Oct 30 01:13:14.856629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856399 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb25f59-b949-474d-a793-8e9714cee19e" containerName="payload-pre-processing" Oct 30 01:13:14.856629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856415 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a1f548-92d1-42e7-9838-53021764e613" containerName="payload-pre-processing" Oct 30 01:13:14.856629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.856427 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2987fc-83e2-49ee-946b-10acfd1668aa" containerName="payload-processing" Oct 30 01:13:14.860191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.860168 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:14.876847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.876822 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:13:14.883638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.883616 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"default-dockercfg-drc54\"" Oct 30 01:13:14.883748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.883707 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"rate-test-model-7f92f6-kserve-self-signed-certs\"" Oct 30 01:13:14.886146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.886122 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"openshift-service-ca.crt\"" Oct 30 01:13:14.886146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.886139 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"kube-root-ca.crt\"" Oct 30 01:13:14.897620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:14.897598 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.011606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.011576 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.011812 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.011793 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vddz\" (UniqueName: \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.011939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.011819 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.012032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.012007 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.012158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.012143 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.012230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.012186 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.038444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.038415 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.125152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.125056 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vddz\" (UniqueName: \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.146983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.146951 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.147969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.147942 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.148631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.148611 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.148813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.148792 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.148991 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.148975 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.149550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.149527 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.155534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.150012 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.155961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.147717 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.189936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.189524 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.190691 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.190664 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.194324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.194300 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vddz\" (UniqueName: \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\") pod \"rate-test-model-7f92f6-kserve-655b9f7-s9k6g\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.476713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.476640 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:15.787831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.787805 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:13:15.788474 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:13:15.788442 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56edaa83_d578_4e8b_a09d_89d53835e246.slice/crio-6b1f4d633d11817259f942a1abd42813c928bf5282b92e1a75fcb10b8ee26fdd WatchSource:0}: Error finding container 6b1f4d633d11817259f942a1abd42813c928bf5282b92e1a75fcb10b8ee26fdd: Status 404 returned error can't find the container with id 6b1f4d633d11817259f942a1abd42813c928bf5282b92e1a75fcb10b8ee26fdd Oct 30 01:13:15.819775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:15.819719 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" event={"ID":"56edaa83-d578-4e8b-a09d-89d53835e246","Type":"ContainerStarted","Data":"6b1f4d633d11817259f942a1abd42813c928bf5282b92e1a75fcb10b8ee26fdd"} Oct 30 01:13:16.310639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.310609 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:16.369817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.369783 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:13:16.406095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.406066 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:13:16.557308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.557277 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:16.695574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.695543 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:16.830199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.830163 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" event={"ID":"56edaa83-d578-4e8b-a09d-89d53835e246","Type":"ContainerStarted","Data":"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c"} Oct 30 01:13:16.842284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.842260 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:16.871962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:16.871913 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" podStartSLOduration=2.871899103 podStartE2EDuration="2.871899103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:16.869242619 +0000 UTC m=+2449.109477733" watchObservedRunningTime="2026-10-30 01:13:16.871899103 +0000 UTC m=+2449.112134215" Oct 30 01:13:17.477561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:17.477528 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:18.101995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.101950 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:18.307981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.305123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:18.507012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.500688 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:13:18.510494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.507540 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.523882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.520511 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.523882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.520567 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zw6\" (UniqueName: \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.523882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.521599 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:13:18.534934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.525107 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a7ca1f-dockercfg-cfqjk\"" Oct 30 01:13:18.534934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.525293 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a7ca1f\"" Oct 30 01:13:18.689883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.688113 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.689883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.688165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zw6\" (UniqueName: \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.699732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.697028 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:13:18.700125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.700051 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.702741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.702706 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.705931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.705905 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:13:18.715702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.715679 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zw6\" (UniqueName: \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\") pod \"payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.789566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.789505 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9fd\" (UniqueName: \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.789566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.789545 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.881906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.881880 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:18.891686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.891659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9fd\" (UniqueName: \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.891808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.891705 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.892513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.892488 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:18.915439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:18.915406 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9fd\" (UniqueName: \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\") pod \"payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:19.043424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.043348 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:19.207376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.207342 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:13:19.373907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.373231 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:13:19.378650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.378619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:19.858757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.858714 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" event={"ID":"0a662095-fe89-48de-b71c-50df497a05dd","Type":"ContainerStarted","Data":"128bb3cf8a2aff61b5f20ba11f69768ce5116a48493991aebd9e1e1fb261c789"} Oct 30 01:13:19.859258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.858793 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:19.859258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.858819 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" event={"ID":"0a662095-fe89-48de-b71c-50df497a05dd","Type":"ContainerStarted","Data":"e5a607f98489627533a1d4f9b000a8b9a868ce62f4cec27633b6d98d55de241d"} Oct 30 01:13:19.860646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.860619 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" event={"ID":"bc520c15-f605-4f32-90dd-5e6d2a5d1e41","Type":"ContainerStarted","Data":"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37"} Oct 30 01:13:19.860774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.860666 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" event={"ID":"bc520c15-f605-4f32-90dd-5e6d2a5d1e41","Type":"ContainerStarted","Data":"393e980e3273d92a8e29ce2f01e5d960819e30fa0d172d4cf258a8c661dab4df"} Oct 30 01:13:19.860774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.860693 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:19.942212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.942152 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" podStartSLOduration=1.942138526 podStartE2EDuration="1.942138526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:19.90008523 +0000 UTC m=+2452.140320345" watchObservedRunningTime="2026-10-30 01:13:19.942138526 +0000 UTC m=+2452.182373638" Oct 30 01:13:19.944265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:19.944220 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" podStartSLOduration=1.944208728 podStartE2EDuration="1.944208728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:19.94067501 +0000 UTC m=+2452.180910126" watchObservedRunningTime="2026-10-30 01:13:19.944208728 +0000 UTC m=+2452.184443849" Oct 30 01:13:21.105731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.105696 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:21.380056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.379945 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" podUID="be1ead2f-a44d-43f9-beed-7f16319447bb" containerName="payload-processing" containerID="cri-o://a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1" gracePeriod=25 Oct 30 01:13:21.420938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.412923 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" podUID="de414aea-a10a-4331-a392-e34b42996819" containerName="payload-pre-processing" containerID="cri-o://8b1928ab01f7c1697e17f0bc310cd68514fe84251bea82fcd278ae07d209cc26" gracePeriod=25 Oct 30 01:13:21.820371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.820339 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:13:21.882290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.882254 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\") pod \"be1ead2f-a44d-43f9-beed-7f16319447bb\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " Oct 30 01:13:21.882517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.882496 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\" (UniqueName: \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\") pod \"be1ead2f-a44d-43f9-beed-7f16319447bb\" (UID: \"be1ead2f-a44d-43f9-beed-7f16319447bb\") " Oct 30 01:13:21.884948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.884880 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume" pod "be1ead2f-a44d-43f9-beed-7f16319447bb" (UID: "be1ead2f-a44d-43f9-beed-7f16319447bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:21.922845 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.922775 5938 generic.go:356] "Generic (PLEG): container finished" podID="be1ead2f-a44d-43f9-beed-7f16319447bb" containerID="a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1" exitCode=0 Oct 30 01:13:21.923120 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.922932 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" event={"ID":"be1ead2f-a44d-43f9-beed-7f16319447bb","Type":"ContainerDied","Data":"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1"} Oct 30 01:13:21.923453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.923262 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" event={"ID":"be1ead2f-a44d-43f9-beed-7f16319447bb","Type":"ContainerDied","Data":"a87357c49909377772a83e1e2525f5a30a8a17443136d6256a2cc8408adf63c8"} Oct 30 01:13:21.923453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.923326 5938 scope.go:122] "RemoveContainer" containerID="a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1" Oct 30 01:13:21.923883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.923751 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9" Oct 30 01:13:21.933201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.933180 5938 generic.go:356] "Generic (PLEG): container finished" podID="de414aea-a10a-4331-a392-e34b42996819" containerID="8b1928ab01f7c1697e17f0bc310cd68514fe84251bea82fcd278ae07d209cc26" exitCode=0 Oct 30 01:13:21.933521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.933347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" event={"ID":"de414aea-a10a-4331-a392-e34b42996819","Type":"ContainerDied","Data":"8b1928ab01f7c1697e17f0bc310cd68514fe84251bea82fcd278ae07d209cc26"} Oct 30 01:13:21.970169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.970127 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh" pod "be1ead2f-a44d-43f9-beed-7f16319447bb" (UID: "be1ead2f-a44d-43f9-beed-7f16319447bb"). InnerVolumeSpecName "kube-api-access-xm8wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:21.981486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.981459 5938 scope.go:122] "RemoveContainer" containerID="a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1" Oct 30 01:13:21.981902 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:13:21.981852 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1\": container with ID starting with a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1 not found: ID does not exist" containerID="a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1" Oct 30 01:13:21.982215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.982186 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1"} err="failed to get container status \"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1\": rpc error: code = NotFound desc = could not find container \"a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1\": container with ID starting with a11df64d673fd804dcfe689cbd2d869571c8a563f7cfa9e07279eeb2aa9a12b1 not found: ID does not exist" Oct 30 01:13:21.984828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.984804 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1ead2f-a44d-43f9-beed-7f16319447bb-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:21.984989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:21.984972 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm8wh\" (UniqueName: \"kubernetes.io/projected/be1ead2f-a44d-43f9-beed-7f16319447bb-kube-api-access-xm8wh\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:22.019708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.019653 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:13:22.085833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.085806 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\") pod \"de414aea-a10a-4331-a392-e34b42996819\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " Oct 30 01:13:22.085998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.085922 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\" (UniqueName: \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\") pod \"de414aea-a10a-4331-a392-e34b42996819\" (UID: \"de414aea-a10a-4331-a392-e34b42996819\") " Oct 30 01:13:22.086469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.086442 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume" pod "de414aea-a10a-4331-a392-e34b42996819" (UID: "de414aea-a10a-4331-a392-e34b42996819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:22.086776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.086754 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de414aea-a10a-4331-a392-e34b42996819-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:22.089213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.089179 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6" pod "de414aea-a10a-4331-a392-e34b42996819" (UID: "de414aea-a10a-4331-a392-e34b42996819"). InnerVolumeSpecName "kube-api-access-6d5s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:22.188593 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.188570 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5s6\" (UniqueName: \"kubernetes.io/projected/de414aea-a10a-4331-a392-e34b42996819-kube-api-access-6d5s6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:22.273936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.273835 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:13:22.279601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.279579 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-84b45b7d48-zntz9"] Oct 30 01:13:22.854346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.854315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:22.943817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.943785 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" Oct 30 01:13:22.944044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.943795 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4" event={"ID":"de414aea-a10a-4331-a392-e34b42996819","Type":"ContainerDied","Data":"d0eae1ffb4639c975b0bf08a89fd04c6f7b5d2f0f0f860f753c91e261bf723d5"} Oct 30 01:13:22.944044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.943921 5938 scope.go:122] "RemoveContainer" containerID="8b1928ab01f7c1697e17f0bc310cd68514fe84251bea82fcd278ae07d209cc26" Oct 30 01:13:22.996681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:22.996649 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:13:23.006008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:23.005977 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-9bcc874ff-j2kz4"] Oct 30 01:13:23.879224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:23.879190 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1ead2f-a44d-43f9-beed-7f16319447bb" path="/var/lib/kubelet/pods/be1ead2f-a44d-43f9-beed-7f16319447bb/volumes" Oct 30 01:13:23.879699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:23.879603 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de414aea-a10a-4331-a392-e34b42996819" path="/var/lib/kubelet/pods/de414aea-a10a-4331-a392-e34b42996819/volumes" Oct 30 01:13:24.107140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.107109 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:24.314154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.313558 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.500774 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502005 5938 cpu_manager.go:414] "containerMap: removing container" podUID="be1ead2f-a44d-43f9-beed-7f16319447bb" containerName="payload-processing" Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502026 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1ead2f-a44d-43f9-beed-7f16319447bb" containerName="payload-processing" Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502123 5938 cpu_manager.go:414] "containerMap: removing container" podUID="de414aea-a10a-4331-a392-e34b42996819" containerName="payload-pre-processing" Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502133 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de414aea-a10a-4331-a392-e34b42996819" containerName="payload-pre-processing" Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502337 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="de414aea-a10a-4331-a392-e34b42996819" containerName="payload-pre-processing" Oct 30 01:13:24.505891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.502356 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1ead2f-a44d-43f9-beed-7f16319447bb" containerName="payload-processing" Oct 30 01:13:24.684499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.684465 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:24.701959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.701933 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.727956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.701825 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:13:24.728223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.728204 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:13:24.730501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.730464 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d8b2c6-dockercfg-2hnpn\"" Oct 30 01:13:24.730735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.730713 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d8b2c6\"" Oct 30 01:13:24.786598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.786560 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:13:24.787045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.786940 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:24.856607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.856535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.856807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.856623 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlbb\" (UniqueName: \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.966939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.958144 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.966939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.958231 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlbb\" (UniqueName: \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.966939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.958372 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:24.966939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.958516 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4cv\" (UniqueName: \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:24.966939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.960067 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:24.983614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:24.982661 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlbb\" (UniqueName: \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\") pod \"payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:25.029955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.029922 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:25.059895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.059832 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.060051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.059997 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4cv\" (UniqueName: \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.060640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.060618 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.095426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.095402 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4cv\" (UniqueName: \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\") pod \"payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.107855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.107476 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.344934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.344130 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:13:25.457095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.456976 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:25.459265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.459194 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:13:25.477306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.477283 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:25.495475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.495451 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:25.981917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.981881 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" event={"ID":"3e0bde88-2f0a-4e90-9362-0b1bf8554541","Type":"ContainerStarted","Data":"5a5ad130705a04686bf27bc630789460ba425a736ebbc3fba97168f91f692882"} Oct 30 01:13:25.982402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.981946 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" event={"ID":"3e0bde88-2f0a-4e90-9362-0b1bf8554541","Type":"ContainerStarted","Data":"02c773b4461814bfed3e76b6e6f404d4e913e1a21579c518158d4f2880fdef7f"} Oct 30 01:13:25.982402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.981970 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:25.983850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.983817 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" event={"ID":"38722356-c23f-4ba3-97df-46c42f68495a","Type":"ContainerStarted","Data":"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef"} Oct 30 01:13:25.984026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.983895 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" event={"ID":"38722356-c23f-4ba3-97df-46c42f68495a","Type":"ContainerStarted","Data":"d3fd18151ff73645825c4a3d1a57554533e2a9ae199e2ad3f01efaf932b30844"} Oct 30 01:13:25.995786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:25.995762 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:13:26.025809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:26.025763 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" podStartSLOduration=2.025751479 podStartE2EDuration="2.025751479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:26.021464932 +0000 UTC m=+2458.261700075" watchObservedRunningTime="2026-10-30 01:13:26.025751479 +0000 UTC m=+2458.265986621" Oct 30 01:13:26.993399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:26.993367 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:27.131982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:27.131944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:28.520233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:28.520169 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" podStartSLOduration=4.520149085 podStartE2EDuration="4.520149085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:26.104025202 +0000 UTC m=+2458.344260350" watchObservedRunningTime="2026-10-30 01:13:28.520149085 +0000 UTC m=+2460.760384198" Oct 30 01:13:28.521888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:28.521850 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:13:28.591833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:28.591797 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:28.845602 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:13:28.845515 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:13:29.846355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:29.846320 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:30.706072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:30.706040 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:30.871461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:30.871432 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:13:30.871878 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:30.871648 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:13:32.327565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:32.327533 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:33.611565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:33.611531 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:13:33.612096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:33.611826 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" containerID="cri-o://fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee" gracePeriod=30 Oct 30 01:13:33.807784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:33.807748 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:34.078367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.078258 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:34.220138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.220102 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:34.383183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.383152 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:34.741028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.740942 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:13:34.746432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.746408 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.773486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.773452 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:13:34.811636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.811613 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cf4567-openshift-default-dockercfg-tsjsw\"" Oct 30 01:13:34.833373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.833348 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:34.843161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843130 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843177 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843292 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843385 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843423 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843523 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843656 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.843721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.843708 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frblj\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.946985 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947063 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947104 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947147 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947301 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947366 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frblj\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947466 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947511 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.947598 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.950889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.948460 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.966883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.954370 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.989593 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.968127 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.989836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.969297 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:34.989987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:34.969504 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.007818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.006492 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.007818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.007120 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.008578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.008551 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.019307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.019139 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.038231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.038162 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frblj\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\") pod \"e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.067570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.067252 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:35.449011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.448973 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:13:35.449938 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:13:35.449909 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7c4d4c_9f6a_44b4_a3bd_d377689b8983.slice/crio-a07ddebb3f13749c45f48723a0ee5cdfbb8e822bfa6276d159a00661d8728012 WatchSource:0}: Error finding container a07ddebb3f13749c45f48723a0ee5cdfbb8e822bfa6276d159a00661d8728012: Status 404 returned error can't find the container with id a07ddebb3f13749c45f48723a0ee5cdfbb8e822bfa6276d159a00661d8728012 Oct 30 01:13:35.453184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.453140 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:35.453313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.453231 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:35.453313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.453280 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:13:35.523084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.523040 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.645475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.645445 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:13:35.650439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.650416 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:35.661313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.661289 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.669456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.669078 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-01215f-dockercfg-zgjrh\"" Oct 30 01:13:35.669456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.669222 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-01215f\"" Oct 30 01:13:35.677911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.674116 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:13:35.861935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.854919 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:13:35.899015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.898984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdwx\" (UniqueName: \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:35.899893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.899852 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:35.928078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.928042 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:35.970638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:35.956559 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:13:36.002140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.002106 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkscv\" (UniqueName: \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.002295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.002151 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdwx\" (UniqueName: \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:36.002380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.002364 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:36.002540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.002508 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.003496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.003472 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:36.028532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.028503 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdwx\" (UniqueName: \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\") pod \"payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:36.104008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.103981 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.104145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.104123 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkscv\" (UniqueName: \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.104665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.104647 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.127217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.127160 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkscv\" (UniqueName: \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\") pod \"payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.144204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.144180 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerStarted","Data":"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168"} Oct 30 01:13:36.144314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.144240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerStarted","Data":"a07ddebb3f13749c45f48723a0ee5cdfbb8e822bfa6276d159a00661d8728012"} Oct 30 01:13:36.192761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.192720 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podStartSLOduration=2.192707666 podStartE2EDuration="2.192707666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:36.187189759 +0000 UTC m=+2468.427424898" watchObservedRunningTime="2026-10-30 01:13:36.192707666 +0000 UTC m=+2468.432942776" Oct 30 01:13:36.273703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.273679 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:36.303166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.302745 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:36.425802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.422468 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:36.615333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.615303 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:13:36.615859 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:13:36.615825 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbff5826_7c67_4771_ac78_ed79fc8b6636.slice/crio-358cbc0464d75783d922d7dd06aa36546d9f595ed84c26aa373f7b347f442405 WatchSource:0}: Error finding container 358cbc0464d75783d922d7dd06aa36546d9f595ed84c26aa373f7b347f442405: Status 404 returned error can't find the container with id 358cbc0464d75783d922d7dd06aa36546d9f595ed84c26aa373f7b347f442405 Oct 30 01:13:36.662024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.661843 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:13:36.671739 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:13:36.671713 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61c8c8e_523f_4de1_8e0e_a9456504f36c.slice/crio-1c65551fda1321239ef99b61a4c6c9f4f5148019ad343e3040bc5a6bb06e479b WatchSource:0}: Error finding container 1c65551fda1321239ef99b61a4c6c9f4f5148019ad343e3040bc5a6bb06e479b: Status 404 returned error can't find the container with id 1c65551fda1321239ef99b61a4c6c9f4f5148019ad343e3040bc5a6bb06e479b Oct 30 01:13:36.994722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:36.994666 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:13:37.068828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.068795 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:37.076922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.076891 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:37.161627 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.161591 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" event={"ID":"f61c8c8e-523f-4de1-8e0e-a9456504f36c","Type":"ContainerStarted","Data":"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840"} Oct 30 01:13:37.161826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.161657 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:37.161826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.161677 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" event={"ID":"f61c8c8e-523f-4de1-8e0e-a9456504f36c","Type":"ContainerStarted","Data":"1c65551fda1321239ef99b61a4c6c9f4f5148019ad343e3040bc5a6bb06e479b"} Oct 30 01:13:37.163331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.163298 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" event={"ID":"fbff5826-7c67-4771-ac78-ed79fc8b6636","Type":"ContainerStarted","Data":"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb"} Oct 30 01:13:37.163452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.163335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" event={"ID":"fbff5826-7c67-4771-ac78-ed79fc8b6636","Type":"ContainerStarted","Data":"358cbc0464d75783d922d7dd06aa36546d9f595ed84c26aa373f7b347f442405"} Oct 30 01:13:37.163526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.163484 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:37.163587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.163528 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:37.164614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.164597 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:13:37.204042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.204000 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" podStartSLOduration=2.20398807 podStartE2EDuration="2.20398807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:37.200148218 +0000 UTC m=+2469.440383330" watchObservedRunningTime="2026-10-30 01:13:37.20398807 +0000 UTC m=+2469.444223197" Oct 30 01:13:37.306881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:37.306753 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" podStartSLOduration=2.306736421 podStartE2EDuration="2.306736421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:37.302016338 +0000 UTC m=+2469.542251452" watchObservedRunningTime="2026-10-30 01:13:37.306736421 +0000 UTC m=+2469.546971538" Oct 30 01:13:38.014444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.014417 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:13:38.173161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.173131 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.679358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.679315 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.817167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.817126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.916095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.916072 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:13:38.965530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965506 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.965667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965542 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.965761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965736 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.965810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965790 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.965902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965835 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.965978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.965923 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:13:38.966049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966000 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.966049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966042 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.966170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966066 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:38.966170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966098 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.966299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966276 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\") pod \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\" (UID: \"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae\") " Oct 30 01:13:38.966366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966292 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:38.966432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966368 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:38.966582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.966561 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:38.967045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.967026 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:38.967045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.967045 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:38.967214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.967055 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:38.967214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.967064 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:38.967214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.967075 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:38.968148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.968125 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:13:38.968254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.968184 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:38.968346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.968328 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:38.968385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:38.968359 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs" pod "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" (UID: "5fcf613f-3ca3-45e4-a4d2-2c948509d5ae"). InnerVolumeSpecName "kube-api-access-w6dbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:39.068296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.068264 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:39.068296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.068292 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:39.068667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.068305 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dbs\" (UniqueName: \"kubernetes.io/projected/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-kube-api-access-w6dbs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:39.068667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.068320 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:39.180816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.180736 5938 generic.go:356] "Generic (PLEG): container finished" podID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerID="fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee" exitCode=0 Oct 30 01:13:39.180816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.180785 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" Oct 30 01:13:39.181106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.180818 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" event={"ID":"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae","Type":"ContainerDied","Data":"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee"} Oct 30 01:13:39.181106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.180915 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp" event={"ID":"5fcf613f-3ca3-45e4-a4d2-2c948509d5ae","Type":"ContainerDied","Data":"b7e01da9677e439345b13940a84eb57e3390b158b683afd633ae2cfe847695f8"} Oct 30 01:13:39.181106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.180949 5938 scope.go:122] "RemoveContainer" containerID="fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee" Oct 30 01:13:39.193739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.193722 5938 scope.go:122] "RemoveContainer" containerID="fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee" Oct 30 01:13:39.194053 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:13:39.194035 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee\": container with ID starting with fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee not found: ID does not exist" containerID="fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee" Oct 30 01:13:39.194118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.194061 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee"} err="failed to get container status \"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee\": rpc error: code = NotFound desc = could not find container \"fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee\": container with ID starting with fe84894b8124b0de39b9b985a7224305afe7f27c28ee2d7366bd610a0d5a09ee not found: ID does not exist" Oct 30 01:13:39.235381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.235358 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:13:39.241032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.241009 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-q86hp"] Oct 30 01:13:39.878438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:39.878405 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" path="/var/lib/kubelet/pods/5fcf613f-3ca3-45e4-a4d2-2c948509d5ae/volumes" Oct 30 01:13:40.215561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:40.215458 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:40.364469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:40.364430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:40.573982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:40.571197 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:42.936586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:42.936555 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:43.550367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.550327 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" podUID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" containerName="main" containerID="cri-o://51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b" gracePeriod=45 Oct 30 01:13:43.839406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.839378 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:13:43.965060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965032 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\" (UniqueName: \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965105 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965163 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965188 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965252 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965372 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\") pod \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\" (UID: \"bf2779dc-475b-45cc-a8c9-c282b24adeb1\") " Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965419 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.965518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965454 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.965949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.965660 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.966122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.966104 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.966122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.966122 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.966268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.966132 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:43.967377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.967354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:13:43.967489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.967391 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "kube-api-access-8vxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:13:43.967489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:43.967472 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs" pod "bf2779dc-475b-45cc-a8c9-c282b24adeb1" (UID: "bf2779dc-475b-45cc-a8c9-c282b24adeb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:13:44.067424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.067370 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxhz\" (UniqueName: \"kubernetes.io/projected/bf2779dc-475b-45cc-a8c9-c282b24adeb1-kube-api-access-8vxhz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:44.067424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.067394 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2779dc-475b-45cc-a8c9-c282b24adeb1-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:44.067424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.067403 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2779dc-475b-45cc-a8c9-c282b24adeb1-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:13:44.225658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.225631 5938 generic.go:356] "Generic (PLEG): container finished" podID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" containerID="51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b" exitCode=0 Oct 30 01:13:44.225770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.225693 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" Oct 30 01:13:44.225770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.225718 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" event={"ID":"bf2779dc-475b-45cc-a8c9-c282b24adeb1","Type":"ContainerDied","Data":"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b"} Oct 30 01:13:44.225770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.225757 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm" event={"ID":"bf2779dc-475b-45cc-a8c9-c282b24adeb1","Type":"ContainerDied","Data":"f410039d6e12e5695af5cab786959f7b89e760f9b3e7796f1fbf854eb4bf33a2"} Oct 30 01:13:44.225915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.225774 5938 scope.go:122] "RemoveContainer" containerID="51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b" Oct 30 01:13:44.237279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.237263 5938 scope.go:122] "RemoveContainer" containerID="51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b" Oct 30 01:13:44.237552 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:13:44.237534 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b\": container with ID starting with 51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b not found: ID does not exist" containerID="51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b" Oct 30 01:13:44.237618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.237559 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b"} err="failed to get container status \"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b\": rpc error: code = NotFound desc = could not find container \"51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b\": container with ID starting with 51426f436fbfc938032f5f5c421cde361eeaf56f33b22d01a03a1ec0ddd5089b not found: ID does not exist" Oct 30 01:13:44.280450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.280426 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:13:44.286301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.286278 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-a5e9ab/auth-test-model-a5e9ab-kserve-6d45c8d685-lgqhm"] Oct 30 01:13:44.388474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.388409 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:13:44.389408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389384 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" Oct 30 01:13:44.389408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389408 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" Oct 30 01:13:44.389619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389465 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" containerName="main" Oct 30 01:13:44.389619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389474 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" containerName="main" Oct 30 01:13:44.389752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389632 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcf613f-3ca3-45e4-a4d2-2c948509d5ae" containerName="istio-proxy" Oct 30 01:13:44.389752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.389649 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" containerName="main" Oct 30 01:13:44.394753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.394731 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.409466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.409446 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a3619c\"/\"openshift-service-ca.crt\"" Oct 30 01:13:44.409721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.409687 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a3619c\"/\"default-dockercfg-t2w8t\"" Oct 30 01:13:44.409904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.409880 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-a3619c\"/\"auth-test-model-a3619c-kserve-self-signed-certs\"" Oct 30 01:13:44.410524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.410504 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-a3619c\"/\"kube-root-ca.crt\"" Oct 30 01:13:44.414213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.414191 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:13:44.477222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477198 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.477363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477251 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.477363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477332 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.477481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477451 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.477522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477511 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.477613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.477594 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtt7s\" (UniqueName: \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.485559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.485535 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:44.578832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.578803 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.578992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.578919 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.578992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.578950 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579010 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579049 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579107 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt7s\" (UniqueName: \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579284 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.579453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.579424 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.581388 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.581363 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.581473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.581438 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.601403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.601382 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt7s\" (UniqueName: \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\") pod \"auth-test-model-a3619c-kserve-59cc76d697-pwz2f\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:44.718936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:44.718152 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:45.051544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.051510 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:13:45.052008 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:13:45.051949 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5df8c94c_0423_4a9a_aba7_e7b0a55eb6b3.slice/crio-f4f47346f5cc6a8b5eaf980a068975627da74c15fa5e57cf8f3b4b4bb5a935f3 WatchSource:0}: Error finding container f4f47346f5cc6a8b5eaf980a068975627da74c15fa5e57cf8f3b4b4bb5a935f3: Status 404 returned error can't find the container with id f4f47346f5cc6a8b5eaf980a068975627da74c15fa5e57cf8f3b4b4bb5a935f3 Oct 30 01:13:45.237292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.237256 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" event={"ID":"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3","Type":"ContainerStarted","Data":"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751"} Oct 30 01:13:45.237292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.237305 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" event={"ID":"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3","Type":"ContainerStarted","Data":"f4f47346f5cc6a8b5eaf980a068975627da74c15fa5e57cf8f3b4b4bb5a935f3"} Oct 30 01:13:45.282731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.282675 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" podStartSLOduration=1.282660096 podStartE2EDuration="1.282660096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:45.279787973 +0000 UTC m=+2477.520023107" watchObservedRunningTime="2026-10-30 01:13:45.282660096 +0000 UTC m=+2477.522895208" Oct 30 01:13:45.346344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.346278 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:45.719344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.719315 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:45.879543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:45.879510 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2779dc-475b-45cc-a8c9-c282b24adeb1" path="/var/lib/kubelet/pods/bf2779dc-475b-45cc-a8c9-c282b24adeb1/volumes" Oct 30 01:13:47.095233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:47.095169 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:48.174511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:48.174468 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:13:48.175022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:48.174543 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:13:48.814437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:48.814405 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:50.770808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:50.770767 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:52.700076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:52.700038 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:54.555003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:54.554967 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:54.719563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:54.719533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:54.732668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:54.732641 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:55.407667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:55.407638 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:13:56.376053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:56.376015 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:58.001719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:58.001692 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:58.836928 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:13:58.836891 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:13:59.608280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:13:59.608241 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:01.565413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:01.563977 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:02.895536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:02.895507 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:03.089524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.089475 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:03.298102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.298002 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:14:03.333629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.333598 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.339138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.339108 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:14:03.364651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.364617 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6d44c2\"" Oct 30 01:14:03.384195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.384168 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6d44c2-dockercfg-2kr24\"" Oct 30 01:14:03.424521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.416142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bzc\" (UniqueName: \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.424959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.424932 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.497921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.476392 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:14:03.499400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.499373 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:14:03.499658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.499644 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.527754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.527717 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bzc\" (UniqueName: \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.528022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.527881 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.528022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.527941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jw7\" (UniqueName: \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.528022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.527978 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.528927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.528900 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.550384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.550327 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bzc\" (UniqueName: \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\") pod \"payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.629463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.629431 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jw7\" (UniqueName: \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.629562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.629471 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.630280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.630256 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.651632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.651607 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jw7\" (UniqueName: \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\") pod \"payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:03.774544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.774521 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:03.820926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:03.820828 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:04.073297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.073221 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:04.087040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.087014 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:14:04.087912 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:14:04.087883 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440e8c82_f2a9_4aae_bf7b_c73e79e440a1.slice/crio-f9fe070edd638f5427fc9b93503de230028a90828dc775fd13a7602ae7427cef WatchSource:0}: Error finding container f9fe070edd638f5427fc9b93503de230028a90828dc775fd13a7602ae7427cef: Status 404 returned error can't find the container with id f9fe070edd638f5427fc9b93503de230028a90828dc775fd13a7602ae7427cef Oct 30 01:14:04.150969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.150937 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:14:04.518914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.518838 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" event={"ID":"ebd06182-e455-455b-bbeb-eb48a512fdd8","Type":"ContainerStarted","Data":"8c63478520a99a57a54e8727255a45610543478d98aca29f67bae7150e92d15a"} Oct 30 01:14:04.519104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.518918 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" event={"ID":"ebd06182-e455-455b-bbeb-eb48a512fdd8","Type":"ContainerStarted","Data":"db1327f32953cf3b3cabb228335f5069311d3323ca3b2d976fd5fdd7d752a785"} Oct 30 01:14:04.519492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.519473 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:04.521128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.521106 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" event={"ID":"440e8c82-f2a9-4aae-bf7b-c73e79e440a1","Type":"ContainerStarted","Data":"ca3cb57f1edc91393107c7b65da44a85253f4f3ff1263d84355bf0f0f4f59064"} Oct 30 01:14:04.521265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.521140 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" event={"ID":"440e8c82-f2a9-4aae-bf7b-c73e79e440a1","Type":"ContainerStarted","Data":"f9fe070edd638f5427fc9b93503de230028a90828dc775fd13a7602ae7427cef"} Oct 30 01:14:04.521265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.521160 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:04.561974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.561932 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" podStartSLOduration=1.561918473 podStartE2EDuration="1.561918473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:04.558802426 +0000 UTC m=+2496.799037543" watchObservedRunningTime="2026-10-30 01:14:04.561918473 +0000 UTC m=+2496.802153585" Oct 30 01:14:04.599053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:04.598802 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" podStartSLOduration=1.5987901820000001 podStartE2EDuration="1.598790182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:04.598210056 +0000 UTC m=+2496.838445172" watchObservedRunningTime="2026-10-30 01:14:04.598790182 +0000 UTC m=+2496.839025295" Oct 30 01:14:05.794095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:05.794056 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:07.696514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:07.696471 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:09.362273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:09.362238 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.481334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:11.481305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.628193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:11.628163 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:13.297150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:13.297119 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:14.965336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:14.965304 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:15.532911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:15.532851 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:14:16.540956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:16.540917 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:14:17.138346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:17.138302 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:18.892802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:18.892769 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:20.598219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:20.598186 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:22.515020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:22.514987 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:24.407401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:24.407371 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:26.252209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:26.252173 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:26.435450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:26.435415 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:26.577640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:26.577564 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:28.311083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:28.309221 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:28.837440 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:28.837403 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:14:29.885817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:29.885783 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:31.453031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.453002 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:14:31.459288 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.459268 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.492243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.492213 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cf4567-serving-cert\"" Oct 30 01:14:31.496937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.496913 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:14:31.521106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.521080 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.521247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.521144 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.521519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.521489 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.521601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.521586 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9q4q\" (UniqueName: \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.578278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.578247 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:31.623196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.623165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.623406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.623388 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.623485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.623445 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9q4q\" (UniqueName: \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.623588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.623575 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.623938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.623921 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.626083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.626062 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.626083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.626076 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.682969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.679647 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9q4q\" (UniqueName: \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\") pod \"maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:31.715184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.715151 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:31.773735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:31.773708 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:32.082635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:32.082609 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:14:32.084227 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:14:32.084206 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698dcf4b_4f94_4f88_a032_7533ff5d8f1e.slice/crio-d4f881330e0dff9aa2e10cd26189b1da4506b2f3cd9c2d2d831ee69d8267dddc WatchSource:0}: Error finding container d4f881330e0dff9aa2e10cd26189b1da4506b2f3cd9c2d2d831ee69d8267dddc: Status 404 returned error can't find the container with id d4f881330e0dff9aa2e10cd26189b1da4506b2f3cd9c2d2d831ee69d8267dddc Oct 30 01:14:32.771783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:32.771696 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" event={"ID":"698dcf4b-4f94-4f88-a032-7533ff5d8f1e","Type":"ContainerStarted","Data":"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42"} Oct 30 01:14:32.771783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:32.771763 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" event={"ID":"698dcf4b-4f94-4f88-a032-7533ff5d8f1e","Type":"ContainerStarted","Data":"d4f881330e0dff9aa2e10cd26189b1da4506b2f3cd9c2d2d831ee69d8267dddc"} Oct 30 01:14:32.772214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:32.771796 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:32.832885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:32.831019 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" podStartSLOduration=1.5024891390000001 podStartE2EDuration="1.831000471s" totalImagesPullingTime="328.511332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:32.829215566 +0000 UTC m=+2525.069450689" watchObservedRunningTime="2026-10-30 01:14:32.831000471 +0000 UTC m=+2525.071235585" Oct 30 01:14:33.586169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:33.586135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.152427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.152390 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.231479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.231439 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:14:35.231703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.231670 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" podUID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" containerName="payload-processing" containerID="cri-o://57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1" gracePeriod=30 Oct 30 01:14:35.370163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.370126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.537457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.537368 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.628310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.628278 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:14:35.659547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.659518 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:14:35.659758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.659731 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" podUID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" containerName="payload-processing" containerID="cri-o://360ad671ef1ddd38e003c566f092b74a51f00c66943b72aad805e2c0246d1ffc" gracePeriod=30 Oct 30 01:14:35.679214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.679182 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.697084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.696234 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:14:35.739023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.737563 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\" (UniqueName: \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\") pod \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " Oct 30 01:14:35.739023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.737830 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\") pod \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\" (UID: \"61a67a99-4b4f-4ca7-8b0c-cda119aded82\") " Oct 30 01:14:35.739547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.739504 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume" pod "61a67a99-4b4f-4ca7-8b0c-cda119aded82" (UID: "61a67a99-4b4f-4ca7-8b0c-cda119aded82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:35.741638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.741603 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls" pod "61a67a99-4b4f-4ca7-8b0c-cda119aded82" (UID: "61a67a99-4b4f-4ca7-8b0c-cda119aded82"). InnerVolumeSpecName "kube-api-access-pjnls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:35.811142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.811104 5938 generic.go:356] "Generic (PLEG): container finished" podID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" containerID="360ad671ef1ddd38e003c566f092b74a51f00c66943b72aad805e2c0246d1ffc" exitCode=0 Oct 30 01:14:35.811396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.811170 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" event={"ID":"7c4fad2d-de82-48f8-8de1-23a1b11414fb","Type":"ContainerDied","Data":"360ad671ef1ddd38e003c566f092b74a51f00c66943b72aad805e2c0246d1ffc"} Oct 30 01:14:35.813411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.813378 5938 generic.go:356] "Generic (PLEG): container finished" podID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" containerID="57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1" exitCode=0 Oct 30 01:14:35.813661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.813432 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" event={"ID":"61a67a99-4b4f-4ca7-8b0c-cda119aded82","Type":"ContainerDied","Data":"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1"} Oct 30 01:14:35.813756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.813489 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" Oct 30 01:14:35.813756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.813701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2" event={"ID":"61a67a99-4b4f-4ca7-8b0c-cda119aded82","Type":"ContainerDied","Data":"e87b59322f8604c20944d13e686b9effc0eec26d418160a133d00a571fa55edc"} Oct 30 01:14:35.813921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.813763 5938 scope.go:122] "RemoveContainer" containerID="57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1" Oct 30 01:14:35.815153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.815130 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" podUID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" containerName="payload-pre-processing" containerID="cri-o://1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16" gracePeriod=30 Oct 30 01:14:35.828776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.828751 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.831589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.831573 5938 scope.go:122] "RemoveContainer" containerID="57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1" Oct 30 01:14:35.831858 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:35.831835 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1\": container with ID starting with 57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1 not found: ID does not exist" containerID="57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1" Oct 30 01:14:35.831992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.831905 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1"} err="failed to get container status \"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1\": rpc error: code = NotFound desc = could not find container \"57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1\": container with ID starting with 57457c4be57c33f5fe044e16a90f8057554d25f4b24c0b715cd7f041e7543af1 not found: ID does not exist" Oct 30 01:14:35.840192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.840170 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjnls\" (UniqueName: \"kubernetes.io/projected/61a67a99-4b4f-4ca7-8b0c-cda119aded82-kube-api-access-pjnls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:35.840276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.840194 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61a67a99-4b4f-4ca7-8b0c-cda119aded82-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:35.950006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.949979 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:14:35.951790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.951764 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:14:35.965124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:35.965094 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-73f667-9d9c68b76-fr9v2"] Oct 30 01:14:36.090950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.083069 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:36.154969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.154933 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\") pod \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " Oct 30 01:14:36.155855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.155828 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\" (UniqueName: \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\") pod \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\" (UID: \"7c4fad2d-de82-48f8-8de1-23a1b11414fb\") " Oct 30 01:14:36.174836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.160746 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume" pod "7c4fad2d-de82-48f8-8de1-23a1b11414fb" (UID: "7c4fad2d-de82-48f8-8de1-23a1b11414fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:36.214886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.166396 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92" pod "7c4fad2d-de82-48f8-8de1-23a1b11414fb" (UID: "7c4fad2d-de82-48f8-8de1-23a1b11414fb"). InnerVolumeSpecName "kube-api-access-m8g92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:36.283814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.283776 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8g92\" (UniqueName: \"kubernetes.io/projected/7c4fad2d-de82-48f8-8de1-23a1b11414fb-kube-api-access-m8g92\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:36.284079 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.284056 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4fad2d-de82-48f8-8de1-23a1b11414fb-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:36.389882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.389609 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:36.442097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.442066 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:14:36.488783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.487359 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\") pod \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " Oct 30 01:14:36.488783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.487495 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\" (UniqueName: \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\") pod \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\" (UID: \"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf\") " Oct 30 01:14:36.489417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.489393 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume" pod "f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" (UID: "f1a8fbce-e322-4ccf-877c-e11bdd79d2cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:36.577002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.576821 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:36.577558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.577519 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9" pod "f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" (UID: "f1a8fbce-e322-4ccf-877c-e11bdd79d2cf"). InnerVolumeSpecName "kube-api-access-ktvr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:36.617893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.608384 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:36.617893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.608426 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvr9\" (UniqueName: \"kubernetes.io/projected/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf-kube-api-access-ktvr9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:36.833021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.832930 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" Oct 30 01:14:36.833021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.832934 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2" event={"ID":"7c4fad2d-de82-48f8-8de1-23a1b11414fb","Type":"ContainerDied","Data":"51ce7cee2c5731732d139200df958462cdad840ceef243a21bcded405979382f"} Oct 30 01:14:36.833021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.833017 5938 scope.go:122] "RemoveContainer" containerID="360ad671ef1ddd38e003c566f092b74a51f00c66943b72aad805e2c0246d1ffc" Oct 30 01:14:36.834737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.834713 5938 generic.go:356] "Generic (PLEG): container finished" podID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" containerID="1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16" exitCode=0 Oct 30 01:14:36.834856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.834765 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" Oct 30 01:14:36.834856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.834802 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" event={"ID":"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf","Type":"ContainerDied","Data":"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16"} Oct 30 01:14:36.834856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.834853 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq" event={"ID":"f1a8fbce-e322-4ccf-877c-e11bdd79d2cf","Type":"ContainerDied","Data":"18f6e6561eb9d82b52391010edc2f4f04d2cc508d8b74e18c45a80bfc4e917c3"} Oct 30 01:14:36.846162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.846123 5938 scope.go:122] "RemoveContainer" containerID="1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16" Oct 30 01:14:36.856776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.856758 5938 scope.go:122] "RemoveContainer" containerID="1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16" Oct 30 01:14:36.857075 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:36.857047 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16\": container with ID starting with 1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16 not found: ID does not exist" containerID="1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16" Oct 30 01:14:36.857152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.857091 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16"} err="failed to get container status \"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16\": rpc error: code = NotFound desc = could not find container \"1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16\": container with ID starting with 1b744ec3264d760f4ae93a5002d77b3844ff4a26f64990080e343f106d2ddb16 not found: ID does not exist" Oct 30 01:14:36.885549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.885530 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:14:36.895798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.895775 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-55a4c7-7cd554dcbc-msrnq"] Oct 30 01:14:36.927837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.927804 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:14:36.932298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:36.932279 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55a4c7-85555b48dd-qlbt2"] Oct 30 01:14:37.053255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:37.053232 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:37.879471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:37.879441 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" path="/var/lib/kubelet/pods/61a67a99-4b4f-4ca7-8b0c-cda119aded82/volumes" Oct 30 01:14:37.879897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:37.879878 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" path="/var/lib/kubelet/pods/7c4fad2d-de82-48f8-8de1-23a1b11414fb/volumes" Oct 30 01:14:37.880285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:37.880271 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" path="/var/lib/kubelet/pods/f1a8fbce-e322-4ccf-877c-e11bdd79d2cf/volumes" Oct 30 01:14:38.333379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:38.333296 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:38.795615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:38.795591 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:14:40.098513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:40.098478 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:40.461496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:40.461458 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:41.805218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:41.805185 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:42.416170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:42.416137 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:14:42.489465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:42.489430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.113477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:43.113443 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.335884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:43.326410 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.562952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:43.562841 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.828215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:43.828095 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.982306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:43.982271 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:44.162128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:44.162089 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:14:44.163623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:44.163588 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" containerID="cri-o://3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3" gracePeriod=30 Oct 30 01:14:44.233684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:44.233648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:45.987937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:45.987900 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:47.791667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:47.791632 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:49.480237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.480209 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:14:49.493045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.493020 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:49.623858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.623823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624028 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624074 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:49.624165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624084 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624161 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624181 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624228 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624295 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624380 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624441 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:49.624507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624460 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:49.624683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624505 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\") pod \"d913e5db-0733-4184-b860-83c44bb85fcc\" (UID: \"d913e5db-0733-4184-b860-83c44bb85fcc\") " Oct 30 01:14:49.624683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624514 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:14:49.624801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.624697 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:49.625346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.625323 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d913e5db-0733-4184-b860-83c44bb85fcc-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.625346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.625345 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.625543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.625354 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.625543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.625363 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.625543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.625373 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.626699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.626669 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "kube-api-access-b6dpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:49.626793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.626708 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:49.626910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.626887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:49.626967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.626948 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo" pod "d913e5db-0733-4184-b860-83c44bb85fcc" (UID: "d913e5db-0733-4184-b860-83c44bb85fcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:14:49.726460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.726404 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d913e5db-0733-4184-b860-83c44bb85fcc-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.726460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.726429 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dpn\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-kube-api-access-b6dpn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.726460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.726439 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d913e5db-0733-4184-b860-83c44bb85fcc-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.726460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.726451 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d913e5db-0733-4184-b860-83c44bb85fcc-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:49.955292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.955263 5938 generic.go:356] "Generic (PLEG): container finished" podID="d913e5db-0733-4184-b860-83c44bb85fcc" containerID="3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3" exitCode=0 Oct 30 01:14:49.955420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.955309 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" Oct 30 01:14:49.955420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.955346 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerDied","Data":"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3"} Oct 30 01:14:49.955420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.955398 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88" event={"ID":"d913e5db-0733-4184-b860-83c44bb85fcc","Type":"ContainerDied","Data":"a961862bc3775e48b4d4a534721e7d6a9dfde7007624c2fb8cf0aa2fde90803e"} Oct 30 01:14:49.955543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.955423 5938 scope.go:122] "RemoveContainer" containerID="3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3" Oct 30 01:14:49.966401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.966360 5938 scope.go:122] "RemoveContainer" containerID="f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412" Oct 30 01:14:49.976397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.976377 5938 scope.go:122] "RemoveContainer" containerID="3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3" Oct 30 01:14:49.976757 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:49.976685 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3\": container with ID starting with 3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3 not found: ID does not exist" containerID="3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3" Oct 30 01:14:49.976757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.976711 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3"} err="failed to get container status \"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3\": rpc error: code = NotFound desc = could not find container \"3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3\": container with ID starting with 3f2094bf5c783a19db391bfa562a5c344ebffd0960e63590dfcf5f239bd46fd3 not found: ID does not exist" Oct 30 01:14:49.976757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.976728 5938 scope.go:122] "RemoveContainer" containerID="f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412" Oct 30 01:14:49.976987 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:49.976969 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412\": container with ID starting with f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412 not found: ID does not exist" containerID="f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412" Oct 30 01:14:49.977033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:49.976994 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412"} err="failed to get container status \"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412\": rpc error: code = NotFound desc = could not find container \"f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412\": container with ID starting with f1c407d765a6cc43f37d61b940beaa45d4f0573436583dbc6e52006544cd9412 not found: ID does not exist" Oct 30 01:14:50.008231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:50.008211 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:14:50.019470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:50.019448 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-55a4c7-openshift-default-759c67ccc7-7qn88"] Oct 30 01:14:51.249040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:51.249008 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.880945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:51.880912 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" path="/var/lib/kubelet/pods/d913e5db-0733-4184-b860-83c44bb85fcc/volumes" Oct 30 01:14:52.949626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:52.949590 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:54.754127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:54.754088 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 01:14:54.754614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:54.754353 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" containerID="cri-o://cccb9561a44836783253d1f7d7059844dea99c7a0cf9e86fa67ba829812e772b" gracePeriod=30 Oct 30 01:14:55.276067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:55.276021 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:56.772846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:56.772814 5938 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:14:56.773242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:56.772878 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:14:56.923770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:56.923733 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:57.446003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.445970 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" podUID="42a80e72-6893-4bb6-a694-1c06cff37416" containerName="main" containerID="cri-o://aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e" gracePeriod=45 Oct 30 01:14:57.739672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.739648 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:14:57.846048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846017 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846214 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846261 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\" (UniqueName: \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:57.846408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846313 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\" (UniqueName: \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846387 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846493 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:57.846628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846533 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\") pod \"42a80e72-6893-4bb6-a694-1c06cff37416\" (UID: \"42a80e72-6893-4bb6-a694-1c06cff37416\") " Oct 30 01:14:57.846836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.846809 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:57.847227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.847204 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:57.847317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.847231 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:57.847317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.847243 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:57.848588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.848562 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:14:57.848709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.848649 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:14:57.848709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.848680 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst" pod "42a80e72-6893-4bb6-a694-1c06cff37416" (UID: "42a80e72-6893-4bb6-a694-1c06cff37416"). InnerVolumeSpecName "kube-api-access-p7cst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:14:57.949012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.948981 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7cst\" (UniqueName: \"kubernetes.io/projected/42a80e72-6893-4bb6-a694-1c06cff37416-kube-api-access-p7cst\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:57.949012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.949008 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a80e72-6893-4bb6-a694-1c06cff37416-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:57.949168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:57.949021 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a80e72-6893-4bb6-a694-1c06cff37416-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:14:58.033350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.033293 5938 generic.go:356] "Generic (PLEG): container finished" podID="42a80e72-6893-4bb6-a694-1c06cff37416" containerID="aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e" exitCode=0 Oct 30 01:14:58.033460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.033348 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" Oct 30 01:14:58.033460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.033368 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" event={"ID":"42a80e72-6893-4bb6-a694-1c06cff37416","Type":"ContainerDied","Data":"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e"} Oct 30 01:14:58.033460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.033412 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n" event={"ID":"42a80e72-6893-4bb6-a694-1c06cff37416","Type":"ContainerDied","Data":"a816cf72604ef850e3e981f99edac62a79f2365099d5cf3d5db3549214d82265"} Oct 30 01:14:58.033460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.033436 5938 scope.go:122] "RemoveContainer" containerID="aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e" Oct 30 01:14:58.044016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.043992 5938 scope.go:122] "RemoveContainer" containerID="aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e" Oct 30 01:14:58.044265 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:58.044246 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e\": container with ID starting with aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e not found: ID does not exist" containerID="aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e" Oct 30 01:14:58.044317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.044271 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e"} err="failed to get container status \"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e\": rpc error: code = NotFound desc = could not find container \"aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e\": container with ID starting with aeb157415d71497d4c39d8b12c3dae6fde0695c7cbba7da254d987097339b11e not found: ID does not exist" Oct 30 01:14:58.084199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.084174 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:14:58.094713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.094688 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/sub-test-model-beeba4-kserve-7995dd78c9-tb68n"] Oct 30 01:14:58.568265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:58.568237 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:58.838271 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:14:58.838195 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:14:59.879205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:14:59.879179 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a80e72-6893-4bb6-a694-1c06cff37416" path="/var/lib/kubelet/pods/42a80e72-6893-4bb6-a694-1c06cff37416/volumes" Oct 30 01:15:00.059158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.059127 5938 generic.go:356] "Generic (PLEG): container finished" podID="a571529c-e955-4307-9501-7c97b832cff8" containerID="cccb9561a44836783253d1f7d7059844dea99c7a0cf9e86fa67ba829812e772b" exitCode=0 Oct 30 01:15:00.059330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.059178 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerDied","Data":"cccb9561a44836783253d1f7d7059844dea99c7a0cf9e86fa67ba829812e772b"} Oct 30 01:15:00.059330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.059231 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" event={"ID":"a571529c-e955-4307-9501-7c97b832cff8","Type":"ContainerDied","Data":"d1b6cfffe814999d8659b8b9facd3d906193b458f2cfd06782e2a2681fe2610d"} Oct 30 01:15:00.059330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.059255 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b6cfffe814999d8659b8b9facd3d906193b458f2cfd06782e2a2681fe2610d" Oct 30 01:15:00.059330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.059272 5938 scope.go:122] "RemoveContainer" containerID="f095fd72cbb94f525a29e07b2e1961e90abc9c2789a7383820e6287682ca3902" Oct 30 01:15:00.066974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.066946 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.190564 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.190686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.190858 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.190932 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.190966 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.191024 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.191070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.191096 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.198425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.191228 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\") pod \"a571529c-e955-4307-9501-7c97b832cff8\" (UID: \"a571529c-e955-4307-9501-7c97b832cff8\") " Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.200042 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.200395 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.200624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201216 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201233 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201249 5938 cpu_manager.go:414] "containerMap: removing container" podUID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201256 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201278 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201284 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201309 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201316 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201331 5938 cpu_manager.go:414] "containerMap: removing container" podUID="42a80e72-6893-4bb6-a694-1c06cff37416" containerName="main" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201337 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a80e72-6893-4bb6-a694-1c06cff37416" containerName="main" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201352 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201359 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201374 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" containerName="payload-pre-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201381 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" containerName="payload-pre-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201514 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201544 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4fad2d-de82-48f8-8de1-23a1b11414fb" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201555 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a67a99-4b4f-4ca7-8b0c-cda119aded82" containerName="payload-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201567 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201582 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201591 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201605 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a80e72-6893-4bb6-a694-1c06cff37416" containerName="main" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201616 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d913e5db-0733-4184-b860-83c44bb85fcc" containerName="istio-proxy" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201628 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a8fbce-e322-4ccf-877c-e11bdd79d2cf" containerName="payload-pre-processing" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.201696 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.202143 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:00.205071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.204128 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:00.209968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.208942 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.214557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.214524 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:00.219389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.215715 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:00.219389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.216790 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.219389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.216809 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a571529c-e955-4307-9501-7c97b832cff8" containerName="istio-proxy" Oct 30 01:15:00.221729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.221602 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.226559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.226530 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:00.260171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.244412 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.260171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.255941 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:00.262912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.262826 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:00.270127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.266138 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:00.288909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.283516 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:00.288909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.283723 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s" pod "a571529c-e955-4307-9501-7c97b832cff8" (UID: "a571529c-e955-4307-9501-7c97b832cff8"). InnerVolumeSpecName "kube-api-access-dgf5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:00.288909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.287324 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.310988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.305642 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:15:00.310988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.306391 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:00.312888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.312849 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.313069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313013 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.313198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313075 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7rz\" (UniqueName: \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.313198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313115 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmct\" (UniqueName: \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.313198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313160 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.313463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313272 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgn4\" (UniqueName: \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.313463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313458 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313474 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313489 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgf5s\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-kube-api-access-dgf5s\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313508 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a571529c-e955-4307-9501-7c97b832cff8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313520 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a571529c-e955-4307-9501-7c97b832cff8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313533 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313546 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313564 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a571529c-e955-4307-9501-7c97b832cff8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.313617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.313575 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a571529c-e955-4307-9501-7c97b832cff8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:00.340651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.340629 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:00.360090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.360066 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.361670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.361651 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:00.367310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.367287 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:00.373790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.373770 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:00.374712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.374334 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.382321 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.378464 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:00.384203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.384163 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:00.384327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.384237 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.401324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.401302 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:00.402714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.402696 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.409390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.409276 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:00.409515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.409398 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.409515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.409427 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:00.414689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.414667 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.414918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.414898 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.415088 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.415069 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7rz\" (UniqueName: \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.415292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.415260 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmct\" (UniqueName: \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.415438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.415423 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.415686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.415670 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgn4\" (UniqueName: \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.416122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.416089 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsmd\" (UniqueName: \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.416283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.416268 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.418287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.418267 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.419239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.419206 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.420449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.420428 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.446931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.445915 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:00.446931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.445959 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:00.446931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.445990 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:00.446931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.446022 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:00.446931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.446139 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.450660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.450637 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgn4\" (UniqueName: \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.454128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.451666 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7rz\" (UniqueName: \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.454254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.451762 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmct\" (UniqueName: \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.454346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.452166 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:00.454462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.454451 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:00.454561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.452228 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.471697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.471673 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:00.519211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.519163 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctww\" (UniqueName: \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.519623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.519599 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.519887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.519847 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.520128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.520108 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.520437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.520414 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5llz\" (UniqueName: \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.520614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.520592 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.520789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.520772 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjkp\" (UniqueName: \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.520985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.520969 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsmd\" (UniqueName: \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.521458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.521436 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9586w\" (UniqueName: \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.521944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.521923 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.522179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.522156 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpmk\" (UniqueName: \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.522400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.522381 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.522640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.522621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.523880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.523845 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nv7\" (UniqueName: \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.524403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.524381 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8qq\" (UniqueName: \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.524661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.524639 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.525061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.523738 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.544961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.544894 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsmd\" (UniqueName: \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.625970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.625943 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nv7\" (UniqueName: \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.626128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626014 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8qq\" (UniqueName: \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.626128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626061 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.626243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626210 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctww\" (UniqueName: \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.626302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626266 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.626364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626302 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.626433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.626491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626469 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5llz\" (UniqueName: \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.626547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626504 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.626602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626542 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjkp\" (UniqueName: \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.626664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626634 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9586w\" (UniqueName: \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.626719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626664 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.626772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626736 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpmk\" (UniqueName: \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.626826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626805 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.626903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.626812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.627124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.627297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627199 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.627460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627434 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.627641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627609 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.627762 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627674 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.627821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.627807 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.649412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.649366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctww\" (UniqueName: \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:00.649748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.649720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nv7\" (UniqueName: \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\") pod \"maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.650147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.650121 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9586w\" (UniqueName: \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\") pod \"maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.650384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.650360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8qq\" (UniqueName: \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\") pod \"maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.650523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.650497 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjkp\" (UniqueName: \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.651320 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.651293 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5llz\" (UniqueName: \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\") pod \"maas-api-key-cleanup-29888715-swzqz\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.651724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.651701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpmk\" (UniqueName: \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\") pod \"maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.656410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.655689 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:00.659934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.659906 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:00.685951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.685922 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:00.691067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.691046 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:00.708090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.708058 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:00.761339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.761100 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:00.766094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.765909 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:00.770802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.770726 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:00.774623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.774572 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:00.814318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.812466 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:00.879200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:00.841651 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:01.085296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.084667 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n" Oct 30 01:15:01.178802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.176536 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 01:15:01.183612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.182266 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-dvw9n"] Oct 30 01:15:01.319172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.319141 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:01.319425 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.319400 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04bcff3_6bb1_47bb_b1ab_0d20743b88e4.slice/crio-cdb153e67a76b6b9381b41db8575b38b1ee581c886489239e382ce7ae23e1626 WatchSource:0}: Error finding container cdb153e67a76b6b9381b41db8575b38b1ee581c886489239e382ce7ae23e1626: Status 404 returned error can't find the container with id cdb153e67a76b6b9381b41db8575b38b1ee581c886489239e382ce7ae23e1626 Oct 30 01:15:01.425519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.425488 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:01.432253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.432228 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:01.465204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.465179 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:01.477049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.477025 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:01.481858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.481837 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:01.485289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.485270 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:01.488677 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.488654 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a290c9e_086d_4d5b_946a_432864322395.slice/crio-b606ba2ba26fd7275b7fe2630c9e580eeed3134cd563326cba9b0fc7c326971f WatchSource:0}: Error finding container b606ba2ba26fd7275b7fe2630c9e580eeed3134cd563326cba9b0fc7c326971f: Status 404 returned error can't find the container with id b606ba2ba26fd7275b7fe2630c9e580eeed3134cd563326cba9b0fc7c326971f Oct 30 01:15:01.489698 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.489668 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b551f_5970_4b36_bc8a_ac547d631b86.slice/crio-14bf9eca2afb49facb0e935048f6cdd6f6cebc173ae4ffd9d92d5b02b1f44fbc WatchSource:0}: Error finding container 14bf9eca2afb49facb0e935048f6cdd6f6cebc173ae4ffd9d92d5b02b1f44fbc: Status 404 returned error can't find the container with id 14bf9eca2afb49facb0e935048f6cdd6f6cebc173ae4ffd9d92d5b02b1f44fbc Oct 30 01:15:01.491397 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.491375 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0e6954_0e5e_4d32_9ded_2b0b557007a4.slice/crio-4d3d357eeeae0fffbff627a2ed68f4d6d57b74122cce1a68804a9eddb77df68f WatchSource:0}: Error finding container 4d3d357eeeae0fffbff627a2ed68f4d6d57b74122cce1a68804a9eddb77df68f: Status 404 returned error can't find the container with id 4d3d357eeeae0fffbff627a2ed68f4d6d57b74122cce1a68804a9eddb77df68f Oct 30 01:15:01.492735 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.492703 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280ba60a_1ddd_4523_a4aa_50cdd719390d.slice/crio-7e685aaa09721c15957d5de72a2bac7de90e874551cb9d43c3d52bb79d528c7e WatchSource:0}: Error finding container 7e685aaa09721c15957d5de72a2bac7de90e874551cb9d43c3d52bb79d528c7e: Status 404 returned error can't find the container with id 7e685aaa09721c15957d5de72a2bac7de90e874551cb9d43c3d52bb79d528c7e Oct 30 01:15:01.519910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.510095 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:01.555539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.555509 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:01.557493 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.557460 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e30fe6a_838f_4021_8cac_1c2b9cf64c2a.slice/crio-0d1ae160b17c7e1583b99d91de0922e495a64493244a646956d10c882285e1dc WatchSource:0}: Error finding container 0d1ae160b17c7e1583b99d91de0922e495a64493244a646956d10c882285e1dc: Status 404 returned error can't find the container with id 0d1ae160b17c7e1583b99d91de0922e495a64493244a646956d10c882285e1dc Oct 30 01:15:01.590021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.589999 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:01.594001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.593980 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:01.707221 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:01.707188 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2772081_0a7b_4a02_a2f3_ddbb04db7b3f.slice/crio-169b743fe24b4d11f23de07dc27c52c5d4f921785e7516ad64f58f401c836466 WatchSource:0}: Error finding container 169b743fe24b4d11f23de07dc27c52c5d4f921785e7516ad64f58f401c836466: Status 404 returned error can't find the container with id 169b743fe24b4d11f23de07dc27c52c5d4f921785e7516ad64f58f401c836466 Oct 30 01:15:01.894485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:01.894453 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a571529c-e955-4307-9501-7c97b832cff8" path="/var/lib/kubelet/pods/a571529c-e955-4307-9501-7c97b832cff8/volumes" Oct 30 01:15:02.099595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.099562 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" event={"ID":"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a","Type":"ContainerStarted","Data":"0d1ae160b17c7e1583b99d91de0922e495a64493244a646956d10c882285e1dc"} Oct 30 01:15:02.101795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.101756 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerStarted","Data":"cefb637f44209aba94c410d060938c77243c188234c134a21e7e12a34f9e390e"} Oct 30 01:15:02.103507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.103485 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerStarted","Data":"ab45413f707d989f8f988896426c5ee725c5b93b9cd8e122d13274788d8ddfb9"} Oct 30 01:15:02.104992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.104969 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerStarted","Data":"169b743fe24b4d11f23de07dc27c52c5d4f921785e7516ad64f58f401c836466"} Oct 30 01:15:02.106601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.106576 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" event={"ID":"2f0e6954-0e5e-4d32-9ded-2b0b557007a4","Type":"ContainerStarted","Data":"4d3d357eeeae0fffbff627a2ed68f4d6d57b74122cce1a68804a9eddb77df68f"} Oct 30 01:15:02.109588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.109564 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerStarted","Data":"14bf9eca2afb49facb0e935048f6cdd6f6cebc173ae4ffd9d92d5b02b1f44fbc"} Oct 30 01:15:02.110917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.110895 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerStarted","Data":"690d810ef1b35cc239958097b92e2afceb137aa079e2627b6f1d13a1cf44732f"} Oct 30 01:15:02.112336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.112316 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerStarted","Data":"42165587a34e5f2d56f03520e7713f45a6bb286b7ed457243867e25e7609dad3"} Oct 30 01:15:02.114044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.114021 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerStarted","Data":"cdb153e67a76b6b9381b41db8575b38b1ee581c886489239e382ce7ae23e1626"} Oct 30 01:15:02.115467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.115437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerStarted","Data":"7e685aaa09721c15957d5de72a2bac7de90e874551cb9d43c3d52bb79d528c7e"} Oct 30 01:15:02.116703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.116682 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" event={"ID":"3a290c9e-086d-4d5b-946a-432864322395","Type":"ContainerStarted","Data":"b606ba2ba26fd7275b7fe2630c9e580eeed3134cd563326cba9b0fc7c326971f"} Oct 30 01:15:02.155208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.155135 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" podStartSLOduration=2.155123066 podStartE2EDuration="2.155123066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.151031182 +0000 UTC m=+2554.391266289" watchObservedRunningTime="2026-10-30 01:15:02.155123066 +0000 UTC m=+2554.395358179" Oct 30 01:15:02.749262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:02.749212 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:03.127953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.127915 5938 generic.go:356] "Generic (PLEG): container finished" podID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerID="10231d93defa7409441b727ac08ca275777bf28244e13025fa47182c59633cd5" exitCode=60 Oct 30 01:15:03.128331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.128011 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerDied","Data":"10231d93defa7409441b727ac08ca275777bf28244e13025fa47182c59633cd5"} Oct 30 01:15:03.128433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.128413 5938 scope.go:122] "RemoveContainer" containerID="10231d93defa7409441b727ac08ca275777bf28244e13025fa47182c59633cd5" Oct 30 01:15:03.131015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.130990 5938 generic.go:356] "Generic (PLEG): container finished" podID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerID="32684f8a7b64471d1513e663a64b7ce63fad87f90d2eaaa16dcab29ca911e99d" exitCode=60 Oct 30 01:15:03.131117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.131101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerDied","Data":"32684f8a7b64471d1513e663a64b7ce63fad87f90d2eaaa16dcab29ca911e99d"} Oct 30 01:15:03.131438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.131421 5938 scope.go:122] "RemoveContainer" containerID="32684f8a7b64471d1513e663a64b7ce63fad87f90d2eaaa16dcab29ca911e99d" Oct 30 01:15:03.135816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.134813 5938 generic.go:356] "Generic (PLEG): container finished" podID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerID="9b4f0cd69b74be31c70b632eb3792884e8b9161dc084575d877402a62c782486" exitCode=60 Oct 30 01:15:03.135816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.134960 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerDied","Data":"9b4f0cd69b74be31c70b632eb3792884e8b9161dc084575d877402a62c782486"} Oct 30 01:15:03.135816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.135118 5938 scope.go:122] "RemoveContainer" containerID="9b4f0cd69b74be31c70b632eb3792884e8b9161dc084575d877402a62c782486" Oct 30 01:15:03.137185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.137162 5938 generic.go:356] "Generic (PLEG): container finished" podID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerID="62097d6566f48e57acfb05b47b947fac8f55b4fa03e665e5610539d83d654086" exitCode=60 Oct 30 01:15:03.137285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.137195 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerDied","Data":"62097d6566f48e57acfb05b47b947fac8f55b4fa03e665e5610539d83d654086"} Oct 30 01:15:03.137511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.137491 5938 scope.go:122] "RemoveContainer" containerID="62097d6566f48e57acfb05b47b947fac8f55b4fa03e665e5610539d83d654086" Oct 30 01:15:03.138902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.138880 5938 generic.go:356] "Generic (PLEG): container finished" podID="3a290c9e-086d-4d5b-946a-432864322395" containerID="75f5bb02237d7d86fabee8756641536178010ec543a14d084a349ef963e81aaf" exitCode=60 Oct 30 01:15:03.139016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.138959 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" event={"ID":"3a290c9e-086d-4d5b-946a-432864322395","Type":"ContainerDied","Data":"75f5bb02237d7d86fabee8756641536178010ec543a14d084a349ef963e81aaf"} Oct 30 01:15:03.139150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.139136 5938 scope.go:122] "RemoveContainer" containerID="75f5bb02237d7d86fabee8756641536178010ec543a14d084a349ef963e81aaf" Oct 30 01:15:03.140776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.140758 5938 generic.go:356] "Generic (PLEG): container finished" podID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerID="8d739affecd996fedb3f00ca06eae8765f8d0c635c1f698533ecb6f7b02b6cc6" exitCode=60 Oct 30 01:15:03.140907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.140833 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" event={"ID":"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a","Type":"ContainerDied","Data":"8d739affecd996fedb3f00ca06eae8765f8d0c635c1f698533ecb6f7b02b6cc6"} Oct 30 01:15:03.141153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.141130 5938 scope.go:122] "RemoveContainer" containerID="8d739affecd996fedb3f00ca06eae8765f8d0c635c1f698533ecb6f7b02b6cc6" Oct 30 01:15:03.142777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.142753 5938 generic.go:356] "Generic (PLEG): container finished" podID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerID="127317d2fd67abd81cd914b92303fe35299c49392159e79c89624587f7d5151c" exitCode=60 Oct 30 01:15:03.142896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.142847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerDied","Data":"127317d2fd67abd81cd914b92303fe35299c49392159e79c89624587f7d5151c"} Oct 30 01:15:03.143287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.143267 5938 scope.go:122] "RemoveContainer" containerID="127317d2fd67abd81cd914b92303fe35299c49392159e79c89624587f7d5151c" Oct 30 01:15:03.147302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.146856 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e73d49a-bd45-4f0c-a545-77d123952274" containerID="3d36ab841a2d8f075d9170a7e0dc871529380d2d0ace79e208abcf8b8a5aa9c6" exitCode=60 Oct 30 01:15:03.147302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.147296 5938 scope.go:122] "RemoveContainer" containerID="3d36ab841a2d8f075d9170a7e0dc871529380d2d0ace79e208abcf8b8a5aa9c6" Oct 30 01:15:03.147622 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.147457 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerDied","Data":"3d36ab841a2d8f075d9170a7e0dc871529380d2d0ace79e208abcf8b8a5aa9c6"} Oct 30 01:15:03.148906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.148886 5938 generic.go:356] "Generic (PLEG): container finished" podID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerID="18255189c6a7480f889fd1a2710a39d8cc2e133c07933463dec9e5e96760b727" exitCode=60 Oct 30 01:15:03.149015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.149000 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerDied","Data":"18255189c6a7480f889fd1a2710a39d8cc2e133c07933463dec9e5e96760b727"} Oct 30 01:15:03.149202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.149185 5938 scope.go:122] "RemoveContainer" containerID="18255189c6a7480f889fd1a2710a39d8cc2e133c07933463dec9e5e96760b727" Oct 30 01:15:03.150614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.150593 5938 generic.go:356] "Generic (PLEG): container finished" podID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerID="673f40bde0cb00b7d0536d604604073e47d04b531f107832e5e979f4559f033b" exitCode=60 Oct 30 01:15:03.150716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.150672 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" event={"ID":"2f0e6954-0e5e-4d32-9ded-2b0b557007a4","Type":"ContainerDied","Data":"673f40bde0cb00b7d0536d604604073e47d04b531f107832e5e979f4559f033b"} Oct 30 01:15:03.150912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.150851 5938 scope.go:122] "RemoveContainer" containerID="673f40bde0cb00b7d0536d604604073e47d04b531f107832e5e979f4559f033b" Oct 30 01:15:03.154012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.153987 5938 generic.go:356] "Generic (PLEG): container finished" podID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerID="2f043668f31f5ab84b7a584f3a26cc6840b0dcae2b4469a44b9512d10833ae6b" exitCode=60 Oct 30 01:15:03.154104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.154023 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerDied","Data":"2f043668f31f5ab84b7a584f3a26cc6840b0dcae2b4469a44b9512d10833ae6b"} Oct 30 01:15:03.154334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:03.154317 5938 scope.go:122] "RemoveContainer" containerID="2f043668f31f5ab84b7a584f3a26cc6840b0dcae2b4469a44b9512d10833ae6b" Oct 30 01:15:04.165643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.165616 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerStarted","Data":"fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480"} Oct 30 01:15:04.168185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.168157 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerStarted","Data":"d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61"} Oct 30 01:15:04.170387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.170364 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerStarted","Data":"1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec"} Oct 30 01:15:04.172466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.172442 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerStarted","Data":"8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139"} Oct 30 01:15:04.174547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.174522 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerStarted","Data":"0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14"} Oct 30 01:15:04.176781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.176759 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerStarted","Data":"41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612"} Oct 30 01:15:04.178976 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.178953 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerStarted","Data":"31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771"} Oct 30 01:15:04.181037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.181017 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerStarted","Data":"c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939"} Oct 30 01:15:04.206047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.206005 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" podStartSLOduration=4.205993897 podStartE2EDuration="4.205993897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.203332395 +0000 UTC m=+2556.443567522" watchObservedRunningTime="2026-10-30 01:15:04.205993897 +0000 UTC m=+2556.446229009" Oct 30 01:15:04.241829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.241785 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" podStartSLOduration=4.241772823 podStartE2EDuration="4.241772823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.241054185 +0000 UTC m=+2556.481289298" watchObservedRunningTime="2026-10-30 01:15:04.241772823 +0000 UTC m=+2556.482007937" Oct 30 01:15:04.284066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.284015 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" podStartSLOduration=4.28399787 podStartE2EDuration="4.28399787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.278508491 +0000 UTC m=+2556.518743607" watchObservedRunningTime="2026-10-30 01:15:04.28399787 +0000 UTC m=+2556.524232986" Oct 30 01:15:04.355194 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.355148 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" podStartSLOduration=4.355135915 podStartE2EDuration="4.355135915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.354354994 +0000 UTC m=+2556.594590119" watchObservedRunningTime="2026-10-30 01:15:04.355135915 +0000 UTC m=+2556.595371028" Oct 30 01:15:04.396163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.396120 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" podStartSLOduration=4.396105613 podStartE2EDuration="4.396105613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.393335415 +0000 UTC m=+2556.633570528" watchObservedRunningTime="2026-10-30 01:15:04.396105613 +0000 UTC m=+2556.636340728" Oct 30 01:15:04.433213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.433175 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" podStartSLOduration=4.43316346 podStartE2EDuration="4.43316346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.431628588 +0000 UTC m=+2556.671863701" watchObservedRunningTime="2026-10-30 01:15:04.43316346 +0000 UTC m=+2556.673398573" Oct 30 01:15:04.471076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.471034 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" podStartSLOduration=4.471022623 podStartE2EDuration="4.471022623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.469526014 +0000 UTC m=+2556.709761128" watchObservedRunningTime="2026-10-30 01:15:04.471022623 +0000 UTC m=+2556.711257735" Oct 30 01:15:04.506816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:04.504276 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:05.193012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.192983 5938 generic.go:356] "Generic (PLEG): container finished" podID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerID="e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87" exitCode=60 Oct 30 01:15:05.193427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.193068 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" event={"ID":"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a","Type":"ContainerDied","Data":"e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87"} Oct 30 01:15:05.193427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.193140 5938 scope.go:122] "RemoveContainer" containerID="8d739affecd996fedb3f00ca06eae8765f8d0c635c1f698533ecb6f7b02b6cc6" Oct 30 01:15:05.193427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.193393 5938 scope.go:122] "RemoveContainer" containerID="e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87" Oct 30 01:15:05.193733 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.193710 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5_odh-ai-gateway-infra(3e30fe6a-838f-4021-8cac-1c2b9cf64c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" Oct 30 01:15:05.195369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.195348 5938 generic.go:356] "Generic (PLEG): container finished" podID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerID="fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480" exitCode=60 Oct 30 01:15:05.195456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.195434 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerDied","Data":"fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480"} Oct 30 01:15:05.195643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.195629 5938 scope.go:122] "RemoveContainer" containerID="fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480" Oct 30 01:15:05.195950 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.195931 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t_odh-ai-gateway-infra(f57ae6db-ee66-407d-bb6f-81f24f5d2c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" Oct 30 01:15:05.197493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.197473 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e73d49a-bd45-4f0c-a545-77d123952274" containerID="d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61" exitCode=60 Oct 30 01:15:05.197659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.197629 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerDied","Data":"d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61"} Oct 30 01:15:05.197775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.197756 5938 scope.go:122] "RemoveContainer" containerID="d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61" Oct 30 01:15:05.198032 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.198011 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-swzqz_odh-ai-gateway-infra(6e73d49a-bd45-4f0c-a545-77d123952274)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" Oct 30 01:15:05.199553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.199533 5938 generic.go:356] "Generic (PLEG): container finished" podID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerID="1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec" exitCode=60 Oct 30 01:15:05.199671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.199612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerDied","Data":"1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec"} Oct 30 01:15:05.199817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.199801 5938 scope.go:122] "RemoveContainer" containerID="1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec" Oct 30 01:15:05.200068 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.200045 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh_odh-ai-gateway-infra(b2772081-0a7b-4a02-a2f3-ddbb04db7b3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" Oct 30 01:15:05.201751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.201728 5938 generic.go:356] "Generic (PLEG): container finished" podID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerID="f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814" exitCode=60 Oct 30 01:15:05.201826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.201805 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" event={"ID":"2f0e6954-0e5e-4d32-9ded-2b0b557007a4","Type":"ContainerDied","Data":"f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814"} Oct 30 01:15:05.202008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.201994 5938 scope.go:122] "RemoveContainer" containerID="f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814" Oct 30 01:15:05.202218 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.202199 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j_odh-ai-gateway-infra(2f0e6954-0e5e-4d32-9ded-2b0b557007a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" Oct 30 01:15:05.207471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.204402 5938 generic.go:356] "Generic (PLEG): container finished" podID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerID="8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139" exitCode=60 Oct 30 01:15:05.207471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.204521 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerDied","Data":"8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139"} Oct 30 01:15:05.207471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.204946 5938 scope.go:122] "RemoveContainer" containerID="8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139" Oct 30 01:15:05.207471 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.205253 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6_odh-ai-gateway-infra(e43b551f-5970-4b36-bc8a-ac547d631b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" Oct 30 01:15:05.208988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.208968 5938 generic.go:356] "Generic (PLEG): container finished" podID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerID="0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14" exitCode=60 Oct 30 01:15:05.209082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.209031 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerDied","Data":"0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14"} Oct 30 01:15:05.209331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.209313 5938 scope.go:122] "RemoveContainer" containerID="0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14" Oct 30 01:15:05.209617 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.209596 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj_odh-ai-gateway-infra(d19a3887-f70f-4dbb-bda6-65c8de4b54fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" Oct 30 01:15:05.211008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.210991 5938 generic.go:356] "Generic (PLEG): container finished" podID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerID="41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612" exitCode=60 Oct 30 01:15:05.211105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.211071 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerDied","Data":"41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612"} Oct 30 01:15:05.211248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.211232 5938 scope.go:122] "RemoveContainer" containerID="41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612" Oct 30 01:15:05.211487 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.211461 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj_odh-ai-gateway-infra(37ce73ed-2f86-4c47-b58c-ee6f418560ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" Oct 30 01:15:05.212837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.212818 5938 generic.go:356] "Generic (PLEG): container finished" podID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerID="31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771" exitCode=60 Oct 30 01:15:05.212968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.212949 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerDied","Data":"31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771"} Oct 30 01:15:05.213126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.213110 5938 scope.go:122] "RemoveContainer" containerID="31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771" Oct 30 01:15:05.213357 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.213337 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm_odh-ai-gateway-infra(f04bcff3-6bb1-47bb-b1ab-0d20743b88e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" Oct 30 01:15:05.214788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.214769 5938 generic.go:356] "Generic (PLEG): container finished" podID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerID="c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939" exitCode=60 Oct 30 01:15:05.214891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.214877 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerDied","Data":"c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939"} Oct 30 01:15:05.215106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.215091 5938 scope.go:122] "RemoveContainer" containerID="c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939" Oct 30 01:15:05.215366 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.215340 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4_odh-ai-gateway-infra(280ba60a-1ddd-4523-a4aa-50cdd719390d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" Oct 30 01:15:05.216779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.216762 5938 generic.go:356] "Generic (PLEG): container finished" podID="3a290c9e-086d-4d5b-946a-432864322395" containerID="60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2" exitCode=60 Oct 30 01:15:05.216852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.216835 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" event={"ID":"3a290c9e-086d-4d5b-946a-432864322395","Type":"ContainerDied","Data":"60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2"} Oct 30 01:15:05.217069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.217055 5938 scope.go:122] "RemoveContainer" containerID="60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2" Oct 30 01:15:05.217266 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:05.217250 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs_odh-ai-gateway-infra(3a290c9e-086d-4d5b-946a-432864322395)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" podUID="3a290c9e-086d-4d5b-946a-432864322395" Oct 30 01:15:05.237483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.237452 5938 scope.go:122] "RemoveContainer" containerID="127317d2fd67abd81cd914b92303fe35299c49392159e79c89624587f7d5151c" Oct 30 01:15:05.252011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.251971 5938 scope.go:122] "RemoveContainer" containerID="3d36ab841a2d8f075d9170a7e0dc871529380d2d0ace79e208abcf8b8a5aa9c6" Oct 30 01:15:05.266475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.266453 5938 scope.go:122] "RemoveContainer" containerID="18255189c6a7480f889fd1a2710a39d8cc2e133c07933463dec9e5e96760b727" Oct 30 01:15:05.284634 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.284613 5938 scope.go:122] "RemoveContainer" containerID="673f40bde0cb00b7d0536d604604073e47d04b531f107832e5e979f4559f033b" Oct 30 01:15:05.300415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.300392 5938 scope.go:122] "RemoveContainer" containerID="2f043668f31f5ab84b7a584f3a26cc6840b0dcae2b4469a44b9512d10833ae6b" Oct 30 01:15:05.314509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.314489 5938 scope.go:122] "RemoveContainer" containerID="10231d93defa7409441b727ac08ca275777bf28244e13025fa47182c59633cd5" Oct 30 01:15:05.328819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.328802 5938 scope.go:122] "RemoveContainer" containerID="32684f8a7b64471d1513e663a64b7ce63fad87f90d2eaaa16dcab29ca911e99d" Oct 30 01:15:05.343658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.343639 5938 scope.go:122] "RemoveContainer" containerID="9b4f0cd69b74be31c70b632eb3792884e8b9161dc084575d877402a62c782486" Oct 30 01:15:05.359106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.359082 5938 scope.go:122] "RemoveContainer" containerID="62097d6566f48e57acfb05b47b947fac8f55b4fa03e665e5610539d83d654086" Oct 30 01:15:05.373035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:05.373019 5938 scope.go:122] "RemoveContainer" containerID="75f5bb02237d7d86fabee8756641536178010ec543a14d084a349ef963e81aaf" Oct 30 01:15:06.030067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.030030 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.094112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.094075 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:15:06.094302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.094281 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" podUID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" containerName="payload-processing" containerID="cri-o://5a5ad130705a04686bf27bc630789460ba425a736ebbc3fba97168f91f692882" gracePeriod=30 Oct 30 01:15:06.126009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.125981 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:15:06.126222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.126196 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" podUID="38722356-c23f-4ba3-97df-46c42f68495a" containerName="payload-pre-processing" containerID="cri-o://4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef" gracePeriod=30 Oct 30 01:15:06.175370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.171993 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.229802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.229774 5938 scope.go:122] "RemoveContainer" containerID="f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814" Oct 30 01:15:06.230277 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.230222 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j_odh-ai-gateway-infra(2f0e6954-0e5e-4d32-9ded-2b0b557007a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" Oct 30 01:15:06.231690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.231666 5938 scope.go:122] "RemoveContainer" containerID="8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139" Oct 30 01:15:06.232050 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.232025 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6_odh-ai-gateway-infra(e43b551f-5970-4b36-bc8a-ac547d631b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" Oct 30 01:15:06.233456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.233437 5938 scope.go:122] "RemoveContainer" containerID="0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14" Oct 30 01:15:06.233853 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.233824 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj_odh-ai-gateway-infra(d19a3887-f70f-4dbb-bda6-65c8de4b54fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" Oct 30 01:15:06.236574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.236309 5938 scope.go:122] "RemoveContainer" containerID="41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612" Oct 30 01:15:06.236642 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.236593 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj_odh-ai-gateway-infra(37ce73ed-2f86-4c47-b58c-ee6f418560ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" Oct 30 01:15:06.239770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.239742 5938 scope.go:122] "RemoveContainer" containerID="31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771" Oct 30 01:15:06.240095 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.240076 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm_odh-ai-gateway-infra(f04bcff3-6bb1-47bb-b1ab-0d20743b88e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" Oct 30 01:15:06.242372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.242306 5938 scope.go:122] "RemoveContainer" containerID="c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939" Oct 30 01:15:06.242652 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.242625 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4_odh-ai-gateway-infra(280ba60a-1ddd-4523-a4aa-50cdd719390d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" Oct 30 01:15:06.245797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.245778 5938 scope.go:122] "RemoveContainer" containerID="60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2" Oct 30 01:15:06.246086 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.246062 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs_odh-ai-gateway-infra(3a290c9e-086d-4d5b-946a-432864322395)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" podUID="3a290c9e-086d-4d5b-946a-432864322395" Oct 30 01:15:06.248780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.248752 5938 scope.go:122] "RemoveContainer" containerID="e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87" Oct 30 01:15:06.249067 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.249041 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5_odh-ai-gateway-infra(3e30fe6a-838f-4021-8cac-1c2b9cf64c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" Oct 30 01:15:06.250144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.250122 5938 scope.go:122] "RemoveContainer" containerID="fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480" Oct 30 01:15:06.250399 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.250375 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t_odh-ai-gateway-infra(f57ae6db-ee66-407d-bb6f-81f24f5d2c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" Oct 30 01:15:06.251973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.251955 5938 scope.go:122] "RemoveContainer" containerID="d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61" Oct 30 01:15:06.252333 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.252311 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-swzqz_odh-ai-gateway-infra(6e73d49a-bd45-4f0c-a545-77d123952274)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" Oct 30 01:15:06.253763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.253744 5938 scope.go:122] "RemoveContainer" containerID="1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec" Oct 30 01:15:06.254028 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:06.254008 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh_odh-ai-gateway-infra(b2772081-0a7b-4a02-a2f3-ddbb04db7b3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" Oct 30 01:15:06.255504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.255484 5938 generic.go:356] "Generic (PLEG): container finished" podID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" containerID="5a5ad130705a04686bf27bc630789460ba425a736ebbc3fba97168f91f692882" exitCode=0 Oct 30 01:15:06.255580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.255509 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" event={"ID":"3e0bde88-2f0a-4e90-9362-0b1bf8554541","Type":"ContainerDied","Data":"5a5ad130705a04686bf27bc630789460ba425a736ebbc3fba97168f91f692882"} Oct 30 01:15:06.327910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.324619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.707079 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.707027 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.716195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.716165 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:15:06.767977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.767941 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:15:06.804825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.804795 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\" (UniqueName: \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\") pod \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " Oct 30 01:15:06.805063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805043 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\") pod \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\" (UID: \"3e0bde88-2f0a-4e90-9362-0b1bf8554541\") " Oct 30 01:15:06.805192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805174 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\") pod \"38722356-c23f-4ba3-97df-46c42f68495a\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " Oct 30 01:15:06.805290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805269 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\" (UniqueName: \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\") pod \"38722356-c23f-4ba3-97df-46c42f68495a\" (UID: \"38722356-c23f-4ba3-97df-46c42f68495a\") " Oct 30 01:15:06.805506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805481 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume" pod "3e0bde88-2f0a-4e90-9362-0b1bf8554541" (UID: "3e0bde88-2f0a-4e90-9362-0b1bf8554541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:06.805626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805604 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume" pod "38722356-c23f-4ba3-97df-46c42f68495a" (UID: "38722356-c23f-4ba3-97df-46c42f68495a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:06.805947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805928 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0bde88-2f0a-4e90-9362-0b1bf8554541-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:06.806010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.805951 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38722356-c23f-4ba3-97df-46c42f68495a-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:06.807241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.807213 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv" pod "3e0bde88-2f0a-4e90-9362-0b1bf8554541" (UID: "3e0bde88-2f0a-4e90-9362-0b1bf8554541"). InnerVolumeSpecName "kube-api-access-hv4cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:06.807671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.807651 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb" pod "38722356-c23f-4ba3-97df-46c42f68495a" (UID: "38722356-c23f-4ba3-97df-46c42f68495a"). InnerVolumeSpecName "kube-api-access-xhlbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:06.860602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.860581 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:06.921018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.911416 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4cv\" (UniqueName: \"kubernetes.io/projected/3e0bde88-2f0a-4e90-9362-0b1bf8554541-kube-api-access-hv4cv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:06.921018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:06.911450 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlbb\" (UniqueName: \"kubernetes.io/projected/38722356-c23f-4ba3-97df-46c42f68495a-kube-api-access-xhlbb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:07.028821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.028776 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.220427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.220342 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.265750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.265721 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" Oct 30 01:15:07.266259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.265761 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6" event={"ID":"3e0bde88-2f0a-4e90-9362-0b1bf8554541","Type":"ContainerDied","Data":"02c773b4461814bfed3e76b6e6f404d4e913e1a21579c518158d4f2880fdef7f"} Oct 30 01:15:07.266259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.265827 5938 scope.go:122] "RemoveContainer" containerID="5a5ad130705a04686bf27bc630789460ba425a736ebbc3fba97168f91f692882" Oct 30 01:15:07.267842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.267816 5938 generic.go:356] "Generic (PLEG): container finished" podID="38722356-c23f-4ba3-97df-46c42f68495a" containerID="4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef" exitCode=0 Oct 30 01:15:07.267965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.267901 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" Oct 30 01:15:07.267965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.267904 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" event={"ID":"38722356-c23f-4ba3-97df-46c42f68495a","Type":"ContainerDied","Data":"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef"} Oct 30 01:15:07.267965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.267961 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp" event={"ID":"38722356-c23f-4ba3-97df-46c42f68495a","Type":"ContainerDied","Data":"d3fd18151ff73645825c4a3d1a57554533e2a9ae199e2ad3f01efaf932b30844"} Oct 30 01:15:07.281778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.281709 5938 scope.go:122] "RemoveContainer" containerID="4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef" Oct 30 01:15:07.301026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.301001 5938 scope.go:122] "RemoveContainer" containerID="4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef" Oct 30 01:15:07.302135 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:07.302112 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef\": container with ID starting with 4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef not found: ID does not exist" containerID="4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef" Oct 30 01:15:07.302220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.302146 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef"} err="failed to get container status \"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef\": rpc error: code = NotFound desc = could not find container \"4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef\": container with ID starting with 4c29b97eded33f0e2b21b90fb2dd6eb4bfcca127669e02afc685e08e2f457aef not found: ID does not exist" Oct 30 01:15:07.326831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.326803 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:15:07.339784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.339759 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8b2c6-855bcf88db-7qrb6"] Oct 30 01:15:07.376418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.376389 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:15:07.380345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.380317 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8b2c6-bc6ddf8dc-j9xsp"] Oct 30 01:15:07.401531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.401495 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.570974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.567207 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:07.881544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.881512 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38722356-c23f-4ba3-97df-46c42f68495a" path="/var/lib/kubelet/pods/38722356-c23f-4ba3-97df-46c42f68495a/volumes" Oct 30 01:15:07.882099 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:07.882083 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" path="/var/lib/kubelet/pods/3e0bde88-2f0a-4e90-9362-0b1bf8554541/volumes" Oct 30 01:15:09.166893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:09.162713 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:10.828935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:10.828896 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:10.980754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:10.980722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:11.208928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:11.206230 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:12.623778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:12.623727 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:12.777708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:12.777676 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:13.225816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:13.225786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:14.148921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.148851 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:14.312579 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.312547 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:14.434995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.434699 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:15:14.435172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.435064 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" containerID="cri-o://5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87" gracePeriod=30 Oct 30 01:15:14.514755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.514715 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:14.985585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.985551 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9"] Oct 30 01:15:14.987143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987114 5938 cpu_manager.go:414] "containerMap: removing container" podUID="38722356-c23f-4ba3-97df-46c42f68495a" containerName="payload-pre-processing" Oct 30 01:15:14.987143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987143 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38722356-c23f-4ba3-97df-46c42f68495a" containerName="payload-pre-processing" Oct 30 01:15:14.987354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987174 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" containerName="payload-processing" Oct 30 01:15:14.987354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987183 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" containerName="payload-processing" Oct 30 01:15:14.987478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987408 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0bde88-2f0a-4e90-9362-0b1bf8554541" containerName="payload-processing" Oct 30 01:15:14.987478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.987435 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="38722356-c23f-4ba3-97df-46c42f68495a" containerName="payload-pre-processing" Oct 30 01:15:14.993315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.993294 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:14.996855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.996831 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:14.996985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.996922 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45mj\" (UniqueName: \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:14.997160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:14.997139 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.008891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.008837 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9"] Oct 30 01:15:15.098265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.098234 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45mj\" (UniqueName: \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.098438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.098397 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.098648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.098632 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.099296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.099277 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.101505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.101480 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:15:15.111556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.111537 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.120462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.120441 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45mj\" (UniqueName: \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.189899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.189840 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:15.311913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.311783 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:15.620464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.620430 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9"] Oct 30 01:15:15.621268 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:15:15.621239 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3be788_5ac5_48b2_bde6_df37809bd3ef.slice/crio-a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1 WatchSource:0}: Error finding container a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1: Status 404 returned error can't find the container with id a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1 Oct 30 01:15:15.869316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.869255 5938 scope.go:122] "RemoveContainer" containerID="c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939" Oct 30 01:15:15.869458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.869400 5938 scope.go:122] "RemoveContainer" containerID="fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480" Oct 30 01:15:15.869680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.869622 5938 scope.go:122] "RemoveContainer" containerID="60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2" Oct 30 01:15:15.869761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.869749 5938 scope.go:122] "RemoveContainer" containerID="41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612" Oct 30 01:15:15.870013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:15.869975 5938 scope.go:122] "RemoveContainer" containerID="31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771" Oct 30 01:15:16.406955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:16.406124 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" event={"ID":"4d3be788-5ac5-48b2-bde6-df37809bd3ef","Type":"ContainerStarted","Data":"bcf95a7c24ef301bea71dd7b06d9e4beab5ba3ff0461bf8caf8e053b1456315a"} Oct 30 01:15:16.406955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:16.406192 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" event={"ID":"4d3be788-5ac5-48b2-bde6-df37809bd3ef","Type":"ContainerStarted","Data":"a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1"} Oct 30 01:15:16.450217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:16.450137 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" podStartSLOduration=2.450117623 podStartE2EDuration="2.450117623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:16.445157625 +0000 UTC m=+2568.685392739" watchObservedRunningTime="2026-10-30 01:15:16.450117623 +0000 UTC m=+2568.690352738" Oct 30 01:15:16.868896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:16.868833 5938 scope.go:122] "RemoveContainer" containerID="0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14" Oct 30 01:15:17.093026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.092382 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:17.419960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.419848 5938 generic.go:356] "Generic (PLEG): container finished" podID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerID="063b93c9f231bf5840f96904957e4de4536194c6a873739138b6bf34a5ecc570" exitCode=60 Oct 30 01:15:17.419960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.419923 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerDied","Data":"063b93c9f231bf5840f96904957e4de4536194c6a873739138b6bf34a5ecc570"} Oct 30 01:15:17.420507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.420001 5938 scope.go:122] "RemoveContainer" containerID="41b3cba479c7930cafcaa09ec93cee5b9a268d3f22f57f29df42e11b82af4612" Oct 30 01:15:17.420599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.420526 5938 scope.go:122] "RemoveContainer" containerID="063b93c9f231bf5840f96904957e4de4536194c6a873739138b6bf34a5ecc570" Oct 30 01:15:17.421851 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:17.421811 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj_odh-ai-gateway-infra(37ce73ed-2f86-4c47-b58c-ee6f418560ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" Oct 30 01:15:17.443406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.443379 5938 generic.go:356] "Generic (PLEG): container finished" podID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerID="79e8cc28ee0e6eb0f96e89228f41b89149b5036f4cef280b2cac496743ad0397" exitCode=60 Oct 30 01:15:17.443528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.443494 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerDied","Data":"79e8cc28ee0e6eb0f96e89228f41b89149b5036f4cef280b2cac496743ad0397"} Oct 30 01:15:17.444146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.444126 5938 scope.go:122] "RemoveContainer" containerID="79e8cc28ee0e6eb0f96e89228f41b89149b5036f4cef280b2cac496743ad0397" Oct 30 01:15:17.444652 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:17.444625 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm_odh-ai-gateway-infra(f04bcff3-6bb1-47bb-b1ab-0d20743b88e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" Oct 30 01:15:17.449226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.449205 5938 generic.go:356] "Generic (PLEG): container finished" podID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerID="e8780dcbef65f534f29d0507c5eb0a02e52351976a5a600e1dda5ae243f8e89e" exitCode=60 Oct 30 01:15:17.449343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.449277 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerDied","Data":"e8780dcbef65f534f29d0507c5eb0a02e52351976a5a600e1dda5ae243f8e89e"} Oct 30 01:15:17.449661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.449646 5938 scope.go:122] "RemoveContainer" containerID="e8780dcbef65f534f29d0507c5eb0a02e52351976a5a600e1dda5ae243f8e89e" Oct 30 01:15:17.450031 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:17.450006 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4_odh-ai-gateway-infra(280ba60a-1ddd-4523-a4aa-50cdd719390d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" Oct 30 01:15:17.450984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.450961 5938 generic.go:356] "Generic (PLEG): container finished" podID="4d3be788-5ac5-48b2-bde6-df37809bd3ef" containerID="bcf95a7c24ef301bea71dd7b06d9e4beab5ba3ff0461bf8caf8e053b1456315a" exitCode=0 Oct 30 01:15:17.451075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.451018 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" event={"ID":"4d3be788-5ac5-48b2-bde6-df37809bd3ef","Type":"ContainerDied","Data":"bcf95a7c24ef301bea71dd7b06d9e4beab5ba3ff0461bf8caf8e053b1456315a"} Oct 30 01:15:17.453421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.453395 5938 generic.go:356] "Generic (PLEG): container finished" podID="3a290c9e-086d-4d5b-946a-432864322395" containerID="9d05461d1deebcd1b85ccd9fac47df4cdf32dd5b0eb92dfd0c324b6ef088f7e0" exitCode=60 Oct 30 01:15:17.453576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.453559 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" event={"ID":"3a290c9e-086d-4d5b-946a-432864322395","Type":"ContainerDied","Data":"9d05461d1deebcd1b85ccd9fac47df4cdf32dd5b0eb92dfd0c324b6ef088f7e0"} Oct 30 01:15:17.453980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.453960 5938 scope.go:122] "RemoveContainer" containerID="9d05461d1deebcd1b85ccd9fac47df4cdf32dd5b0eb92dfd0c324b6ef088f7e0" Oct 30 01:15:17.454428 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:17.454402 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs_odh-ai-gateway-infra(3a290c9e-086d-4d5b-946a-432864322395)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" podUID="3a290c9e-086d-4d5b-946a-432864322395" Oct 30 01:15:17.455997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.455975 5938 generic.go:356] "Generic (PLEG): container finished" podID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerID="cf5498a67ba741de48753bf299d08bbf44027c6d8ff56dcf0baa50cb3e5aa93b" exitCode=60 Oct 30 01:15:17.456108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.456048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerDied","Data":"cf5498a67ba741de48753bf299d08bbf44027c6d8ff56dcf0baa50cb3e5aa93b"} Oct 30 01:15:17.456374 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.456359 5938 scope.go:122] "RemoveContainer" containerID="cf5498a67ba741de48753bf299d08bbf44027c6d8ff56dcf0baa50cb3e5aa93b" Oct 30 01:15:17.456576 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:17.456560 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t_odh-ai-gateway-infra(f57ae6db-ee66-407d-bb6f-81f24f5d2c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" Oct 30 01:15:17.470227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.470208 5938 scope.go:122] "RemoveContainer" containerID="31b7f25144ab5ccf3b65253637cfc434b389a270b9522b6cf897677c7b247771" Oct 30 01:15:17.503607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.503583 5938 scope.go:122] "RemoveContainer" containerID="c9a53b1474313ad70800df8afa001f3cfdea73e6ff254092f596064bf6d04939" Oct 30 01:15:17.536977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.536952 5938 scope.go:122] "RemoveContainer" containerID="60fbcfc2160f6d611f8e75b3d9e9461d145ec881a8738c382ff1993f5cd36ca2" Oct 30 01:15:17.567205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.567184 5938 scope.go:122] "RemoveContainer" containerID="fb1d4d694139ff366af9086a27459104d47181215f6616991781f668ed1f9480" Oct 30 01:15:17.875137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:17.875106 5938 scope.go:122] "RemoveContainer" containerID="8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139" Oct 30 01:15:18.471656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.471574 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerStarted","Data":"226b3f69f0d262110ac3ca70fcf09a1a05efe2034d36f413483b8ac08f6ece7c"} Oct 30 01:15:18.472450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.472429 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:18.473691 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.473670 5938 generic.go:356] "Generic (PLEG): container finished" podID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerID="d1681a8ca929e66933e2812cd96c293c7cdb9b59299d840fb6e640c85fdbf01d" exitCode=60 Oct 30 01:15:18.473799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.473733 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerDied","Data":"d1681a8ca929e66933e2812cd96c293c7cdb9b59299d840fb6e640c85fdbf01d"} Oct 30 01:15:18.473799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.473789 5938 scope.go:122] "RemoveContainer" containerID="0a9a82f26b52ec290918c521e647a3e43d27f2da7a557785f2e047490d0d4d14" Oct 30 01:15:18.474128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.474107 5938 scope.go:122] "RemoveContainer" containerID="d1681a8ca929e66933e2812cd96c293c7cdb9b59299d840fb6e640c85fdbf01d" Oct 30 01:15:18.474392 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:18.474364 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj_odh-ai-gateway-infra(d19a3887-f70f-4dbb-bda6-65c8de4b54fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" Oct 30 01:15:18.578007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.577976 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:18.604723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.604687 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:18.673664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.673585 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:18.712607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.708358 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:18.922296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.922261 5938 scope.go:122] "RemoveContainer" containerID="e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87" Oct 30 01:15:18.925822 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.925794 5938 scope.go:122] "RemoveContainer" containerID="1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec" Oct 30 01:15:18.926794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.926776 5938 scope.go:122] "RemoveContainer" containerID="f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814" Oct 30 01:15:18.931748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.930819 5938 scope.go:122] "RemoveContainer" containerID="d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61" Oct 30 01:15:18.944894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:18.944852 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:19.020925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.019568 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:19.023576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.022356 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\" (UniqueName: \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\") pod \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " Oct 30 01:15:19.023576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.022509 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\") pod \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\" (UID: \"37ce73ed-2f86-4c47-b58c-ee6f418560ec\") " Oct 30 01:15:19.023576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.022616 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\" (UniqueName: \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\") pod \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " Oct 30 01:15:19.023576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.022686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\") pod \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " Oct 30 01:15:19.023576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.022722 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\") pod \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\" (UID: \"4d3be788-5ac5-48b2-bde6-df37809bd3ef\") " Oct 30 01:15:19.024615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.024169 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca" pod "37ce73ed-2f86-4c47-b58c-ee6f418560ec" (UID: "37ce73ed-2f86-4c47-b58c-ee6f418560ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.025389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.025362 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca" pod "4d3be788-5ac5-48b2-bde6-df37809bd3ef" (UID: "4d3be788-5ac5-48b2-bde6-df37809bd3ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.025890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.025591 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp" pod "37ce73ed-2f86-4c47-b58c-ee6f418560ec" (UID: "37ce73ed-2f86-4c47-b58c-ee6f418560ec"). InnerVolumeSpecName "kube-api-access-kqjkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.028340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.028164 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca" pod "4d3be788-5ac5-48b2-bde6-df37809bd3ef" (UID: "4d3be788-5ac5-48b2-bde6-df37809bd3ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:15:19.028980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.028543 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ce73ed-2f86-4c47-b58c-ee6f418560ec-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.038698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.038042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj" pod "4d3be788-5ac5-48b2-bde6-df37809bd3ef" (UID: "4d3be788-5ac5-48b2-bde6-df37809bd3ef"). InnerVolumeSpecName "kube-api-access-j45mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.130882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.130760 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45mj\" (UniqueName: \"kubernetes.io/projected/4d3be788-5ac5-48b2-bde6-df37809bd3ef-kube-api-access-j45mj\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.130882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.130800 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.130882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.130819 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d3be788-5ac5-48b2-bde6-df37809bd3ef-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.130882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.130835 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjkp\" (UniqueName: \"kubernetes.io/projected/37ce73ed-2f86-4c47-b58c-ee6f418560ec-kube-api-access-kqjkp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.219450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.219422 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:19.236991 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.235381 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\" (UniqueName: \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\") pod \"280ba60a-1ddd-4523-a4aa-50cdd719390d\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " Oct 30 01:15:19.236991 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.235771 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\") pod \"280ba60a-1ddd-4523-a4aa-50cdd719390d\" (UID: \"280ba60a-1ddd-4523-a4aa-50cdd719390d\") " Oct 30 01:15:19.238522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.238474 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca" pod "280ba60a-1ddd-4523-a4aa-50cdd719390d" (UID: "280ba60a-1ddd-4523-a4aa-50cdd719390d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.242903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.242822 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq" pod "280ba60a-1ddd-4523-a4aa-50cdd719390d" (UID: "280ba60a-1ddd-4523-a4aa-50cdd719390d"). InnerVolumeSpecName "kube-api-access-jj8qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.252280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.252254 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:19.279205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.273283 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:19.282463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.282382 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:19.337629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.337593 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\") pod \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " Oct 30 01:15:19.337809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.337671 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\") pod \"3a290c9e-086d-4d5b-946a-432864322395\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " Oct 30 01:15:19.337941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.337921 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\") pod \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " Oct 30 01:15:19.337988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.337971 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\" (UniqueName: \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\") pod \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\" (UID: \"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4\") " Oct 30 01:15:19.338149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.338133 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\" (UniqueName: \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\") pod \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\" (UID: \"f57ae6db-ee66-407d-bb6f-81f24f5d2c92\") " Oct 30 01:15:19.338230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.338216 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\" (UniqueName: \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\") pod \"3a290c9e-086d-4d5b-946a-432864322395\" (UID: \"3a290c9e-086d-4d5b-946a-432864322395\") " Oct 30 01:15:19.338996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.338974 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8qq\" (UniqueName: \"kubernetes.io/projected/280ba60a-1ddd-4523-a4aa-50cdd719390d-kube-api-access-jj8qq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.339045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.339003 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ba60a-1ddd-4523-a4aa-50cdd719390d-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.340110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.340063 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca" pod "3a290c9e-086d-4d5b-946a-432864322395" (UID: "3a290c9e-086d-4d5b-946a-432864322395"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.340110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.340096 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca" pod "f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" (UID: "f04bcff3-6bb1-47bb-b1ab-0d20743b88e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.340915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.340837 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca" pod "f57ae6db-ee66-407d-bb6f-81f24f5d2c92" (UID: "f57ae6db-ee66-407d-bb6f-81f24f5d2c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.365287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.365250 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:19.399026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.397402 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct" pod "3a290c9e-086d-4d5b-946a-432864322395" (UID: "3a290c9e-086d-4d5b-946a-432864322395"). InnerVolumeSpecName "kube-api-access-5bmct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.410483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.410445 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz" pod "f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" (UID: "f04bcff3-6bb1-47bb-b1ab-0d20743b88e4"). InnerVolumeSpecName "kube-api-access-2t7rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.410697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.410498 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww" pod "f57ae6db-ee66-407d-bb6f-81f24f5d2c92" (UID: "f57ae6db-ee66-407d-bb6f-81f24f5d2c92"). InnerVolumeSpecName "kube-api-access-vctww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.443969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.443765 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctww\" (UniqueName: \"kubernetes.io/projected/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-kube-api-access-vctww\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.444149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.443982 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmct\" (UniqueName: \"kubernetes.io/projected/3a290c9e-086d-4d5b-946a-432864322395-kube-api-access-5bmct\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.444149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.444001 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ae6db-ee66-407d-bb6f-81f24f5d2c92-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.444149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.444030 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a290c9e-086d-4d5b-946a-432864322395-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.444149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.444046 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.444424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.444066 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7rz\" (UniqueName: \"kubernetes.io/projected/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4-kube-api-access-2t7rz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.492666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.492639 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" Oct 30 01:15:19.493127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.492633 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9" event={"ID":"4d3be788-5ac5-48b2-bde6-df37809bd3ef","Type":"ContainerDied","Data":"a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1"} Oct 30 01:15:19.493127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.492795 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3142138944ccbd8fd741d3491657e57b435b3114f65367c6bc2555bd9768db1" Oct 30 01:15:19.494348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.494331 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" Oct 30 01:15:19.494465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.494335 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs" event={"ID":"3a290c9e-086d-4d5b-946a-432864322395","Type":"ContainerDied","Data":"b606ba2ba26fd7275b7fe2630c9e580eeed3134cd563326cba9b0fc7c326971f"} Oct 30 01:15:19.494465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.494413 5938 scope.go:122] "RemoveContainer" containerID="9d05461d1deebcd1b85ccd9fac47df4cdf32dd5b0eb92dfd0c324b6ef088f7e0" Oct 30 01:15:19.498181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.498156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" event={"ID":"f57ae6db-ee66-407d-bb6f-81f24f5d2c92","Type":"ContainerDied","Data":"cefb637f44209aba94c410d060938c77243c188234c134a21e7e12a34f9e390e"} Oct 30 01:15:19.498271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.498167 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t" Oct 30 01:15:19.500680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.500657 5938 generic.go:356] "Generic (PLEG): container finished" podID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerID="226b3f69f0d262110ac3ca70fcf09a1a05efe2034d36f413483b8ac08f6ece7c" exitCode=60 Oct 30 01:15:19.500782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.500728 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerDied","Data":"226b3f69f0d262110ac3ca70fcf09a1a05efe2034d36f413483b8ac08f6ece7c"} Oct 30 01:15:19.501550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.501531 5938 scope.go:122] "RemoveContainer" containerID="226b3f69f0d262110ac3ca70fcf09a1a05efe2034d36f413483b8ac08f6ece7c" Oct 30 01:15:19.501954 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:19.501905 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6_odh-ai-gateway-infra(e43b551f-5970-4b36-bc8a-ac547d631b86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" Oct 30 01:15:19.504801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.504778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" event={"ID":"37ce73ed-2f86-4c47-b58c-ee6f418560ec","Type":"ContainerDied","Data":"42165587a34e5f2d56f03520e7713f45a6bb286b7ed457243867e25e7609dad3"} Oct 30 01:15:19.504915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.504788 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj" Oct 30 01:15:19.507530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.507511 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" Oct 30 01:15:19.507700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.507536 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm" event={"ID":"f04bcff3-6bb1-47bb-b1ab-0d20743b88e4","Type":"ContainerDied","Data":"cdb153e67a76b6b9381b41db8575b38b1ee581c886489239e382ce7ae23e1626"} Oct 30 01:15:19.509355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.509333 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" event={"ID":"280ba60a-1ddd-4523-a4aa-50cdd719390d","Type":"ContainerDied","Data":"7e685aaa09721c15957d5de72a2bac7de90e874551cb9d43c3d52bb79d528c7e"} Oct 30 01:15:19.509470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.509358 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4" Oct 30 01:15:19.523510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.523486 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:19.564550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.564525 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:19.594964 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.594927 5938 scope.go:122] "RemoveContainer" containerID="cf5498a67ba741de48753bf299d08bbf44027c6d8ff56dcf0baa50cb3e5aa93b" Oct 30 01:15:19.620057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.619937 5938 scope.go:122] "RemoveContainer" containerID="8a9456020a0b7e9aca2a8b93505134688a9e401fb8901234287f2298e473c139" Oct 30 01:15:19.654020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.652435 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:19.654449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.654428 5938 scope.go:122] "RemoveContainer" containerID="063b93c9f231bf5840f96904957e4de4536194c6a873739138b6bf34a5ecc570" Oct 30 01:15:19.669452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.669429 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-52650f-29888715-59sh4"] Oct 30 01:15:19.689226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.689201 5938 scope.go:122] "RemoveContainer" containerID="79e8cc28ee0e6eb0f96e89228f41b89149b5036f4cef280b2cac496743ad0397" Oct 30 01:15:19.704309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.704284 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:19.712251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.712225 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a3619c-29888715-7jsqm"] Oct 30 01:15:19.748717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.748657 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:19.749989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.749942 5938 scope.go:122] "RemoveContainer" containerID="e8780dcbef65f534f29d0507c5eb0a02e52351976a5a600e1dda5ae243f8e89e" Oct 30 01:15:19.755689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.755668 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-73435a-29888715-kjcrs"] Oct 30 01:15:19.783058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.783036 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:19.791410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.791388 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca5363-29888715-r7dbj"] Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.837031 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.859693 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.867480 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8e0f66-29888715-dsh4t"] Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.878475 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.880933 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" path="/var/lib/kubelet/pods/280ba60a-1ddd-4523-a4aa-50cdd719390d/volumes" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.881562 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" path="/var/lib/kubelet/pods/37ce73ed-2f86-4c47-b58c-ee6f418560ec/volumes" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.882140 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a290c9e-086d-4d5b-946a-432864322395" path="/var/lib/kubelet/pods/3a290c9e-086d-4d5b-946a-432864322395/volumes" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.883112 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" path="/var/lib/kubelet/pods/f04bcff3-6bb1-47bb-b1ab-0d20743b88e4/volumes" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.883657 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" path="/var/lib/kubelet/pods/f57ae6db-ee66-407d-bb6f-81f24f5d2c92/volumes" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952045 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\") pod \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952190 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952331 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\" (UniqueName: \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\") pod \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\" (UID: \"d19a3887-f70f-4dbb-bda6-65c8de4b54fd\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952379 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca" pod "d19a3887-f70f-4dbb-bda6-65c8de4b54fd" (UID: "d19a3887-f70f-4dbb-bda6-65c8de4b54fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952490 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952528 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952601 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952637 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952657 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952706 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.952814 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\") pod \"34964578-4b4e-4361-aa9b-cc61ce9f9110\" (UID: \"34964578-4b4e-4361-aa9b-cc61ce9f9110\") " Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.953594 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.953617 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.953634 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34964578-4b4e-4361-aa9b-cc61ce9f9110-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.953897 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.954096 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.954283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.955726 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w" pod "d19a3887-f70f-4dbb-bda6-65c8de4b54fd" (UID: "d19a3887-f70f-4dbb-bda6-65c8de4b54fd"). InnerVolumeSpecName "kube-api-access-9586w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.956520 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "kube-api-access-cdjnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.956573 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.958405 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.967267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:19.958635 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo" pod "34964578-4b4e-4361-aa9b-cc61ce9f9110" (UID: "34964578-4b4e-4361-aa9b-cc61ce9f9110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:20.055572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055545 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055572 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdjnz\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-kube-api-access-cdjnz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055582 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055592 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055600 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055607 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34964578-4b4e-4361-aa9b-cc61ce9f9110-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055616 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34964578-4b4e-4361-aa9b-cc61ce9f9110-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.055778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.055625 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9586w\" (UniqueName: \"kubernetes.io/projected/d19a3887-f70f-4dbb-bda6-65c8de4b54fd-kube-api-access-9586w\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.520097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.520072 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" Oct 30 01:15:20.520097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.520086 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj" event={"ID":"d19a3887-f70f-4dbb-bda6-65c8de4b54fd","Type":"ContainerDied","Data":"690d810ef1b35cc239958097b92e2afceb137aa079e2627b6f1d13a1cf44732f"} Oct 30 01:15:20.520540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.520148 5938 scope.go:122] "RemoveContainer" containerID="d1681a8ca929e66933e2812cd96c293c7cdb9b59299d840fb6e640c85fdbf01d" Oct 30 01:15:20.526528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.526496 5938 generic.go:356] "Generic (PLEG): container finished" podID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerID="9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168" exitCode=0 Oct 30 01:15:20.526657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.526565 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerDied","Data":"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168"} Oct 30 01:15:20.527057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.527035 5938 scope.go:122] "RemoveContainer" containerID="9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168" Oct 30 01:15:20.529272 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.529091 5938 generic.go:356] "Generic (PLEG): container finished" podID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerID="fbf0894575e11e03cba68d6c22c25b1f0ea1cd5b4b09c5d3b00d4e647322a207" exitCode=60 Oct 30 01:15:20.529272 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.529165 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" event={"ID":"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a","Type":"ContainerDied","Data":"fbf0894575e11e03cba68d6c22c25b1f0ea1cd5b4b09c5d3b00d4e647322a207"} Oct 30 01:15:20.529589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.529568 5938 scope.go:122] "RemoveContainer" containerID="fbf0894575e11e03cba68d6c22c25b1f0ea1cd5b4b09c5d3b00d4e647322a207" Oct 30 01:15:20.529929 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.529907 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5_odh-ai-gateway-infra(3e30fe6a-838f-4021-8cac-1c2b9cf64c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" Oct 30 01:15:20.533822 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.533666 5938 scope.go:122] "RemoveContainer" containerID="e6dd9843d5db1e3160fd0033fffa1e6330354b5764c6780305c28d4fd54d3e87" Oct 30 01:15:20.534007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.533982 5938 generic.go:356] "Generic (PLEG): container finished" podID="6e73d49a-bd45-4f0c-a545-77d123952274" containerID="516cd14a840aca2a613057da00aedc8083f15d6cc461bd22944481bee775d6cb" exitCode=60 Oct 30 01:15:20.534073 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.534053 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerDied","Data":"516cd14a840aca2a613057da00aedc8083f15d6cc461bd22944481bee775d6cb"} Oct 30 01:15:20.534579 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.534542 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:20.534690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.534566 5938 scope.go:122] "RemoveContainer" containerID="516cd14a840aca2a613057da00aedc8083f15d6cc461bd22944481bee775d6cb" Oct 30 01:15:20.534690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.534631 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:20.534844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.534699 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:15:20.535058 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.535029 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-swzqz_odh-ai-gateway-infra(6e73d49a-bd45-4f0c-a545-77d123952274)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" Oct 30 01:15:20.536382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.536365 5938 generic.go:356] "Generic (PLEG): container finished" podID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerID="06f1f7c3f7a9bf9b9292d4516a7d3874d2af1a1e066d42d3d7b1a924bdd73803" exitCode=60 Oct 30 01:15:20.536495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.536442 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerDied","Data":"06f1f7c3f7a9bf9b9292d4516a7d3874d2af1a1e066d42d3d7b1a924bdd73803"} Oct 30 01:15:20.536790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.536771 5938 scope.go:122] "RemoveContainer" containerID="06f1f7c3f7a9bf9b9292d4516a7d3874d2af1a1e066d42d3d7b1a924bdd73803" Oct 30 01:15:20.537101 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.537075 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh_odh-ai-gateway-infra(b2772081-0a7b-4a02-a2f3-ddbb04db7b3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" Oct 30 01:15:20.538522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.538499 5938 generic.go:356] "Generic (PLEG): container finished" podID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerID="5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87" exitCode=0 Oct 30 01:15:20.538636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.538548 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" Oct 30 01:15:20.538636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.538588 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerDied","Data":"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87"} Oct 30 01:15:20.538760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.538639 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g" event={"ID":"34964578-4b4e-4361-aa9b-cc61ce9f9110","Type":"ContainerDied","Data":"d1ae4c3a49b43fbeb6ec108a4c95c9d6a6b7678011c0a2d6b3a8b34c63629d8b"} Oct 30 01:15:20.540905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.540840 5938 generic.go:356] "Generic (PLEG): container finished" podID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerID="e9bfdfbba868635418367a1d8d1e28703706d7dd87552616d9d2d4925aa50579" exitCode=60 Oct 30 01:15:20.540905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.540891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" event={"ID":"2f0e6954-0e5e-4d32-9ded-2b0b557007a4","Type":"ContainerDied","Data":"e9bfdfbba868635418367a1d8d1e28703706d7dd87552616d9d2d4925aa50579"} Oct 30 01:15:20.541273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.541255 5938 scope.go:122] "RemoveContainer" containerID="e9bfdfbba868635418367a1d8d1e28703706d7dd87552616d9d2d4925aa50579" Oct 30 01:15:20.541543 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.541522 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j_odh-ai-gateway-infra(2f0e6954-0e5e-4d32-9ded-2b0b557007a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" Oct 30 01:15:20.562042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.562016 5938 scope.go:122] "RemoveContainer" containerID="d1d66a35c28ccda2b8b0f8fb27d105808b860de2d86c3da45f73bf7f1726cd61" Oct 30 01:15:20.580443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.580424 5938 scope.go:122] "RemoveContainer" containerID="1e02102bd64d12788fa802970cd75e676020008ff44aa46fe137ff8cb1bfeeec" Oct 30 01:15:20.595166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.594731 5938 scope.go:122] "RemoveContainer" containerID="5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87" Oct 30 01:15:20.622269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.622245 5938 scope.go:122] "RemoveContainer" containerID="0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15" Oct 30 01:15:20.635608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.635589 5938 scope.go:122] "RemoveContainer" containerID="5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87" Oct 30 01:15:20.636277 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.635950 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87\": container with ID starting with 5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87 not found: ID does not exist" containerID="5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87" Oct 30 01:15:20.636277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.635988 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87"} err="failed to get container status \"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87\": rpc error: code = NotFound desc = could not find container \"5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87\": container with ID starting with 5bd8ca97a06822f9234d9877eb3b45baef8d27c5f3ad2634af67fb52cf201d87 not found: ID does not exist" Oct 30 01:15:20.636277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.636013 5938 scope.go:122] "RemoveContainer" containerID="0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15" Oct 30 01:15:20.637038 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:20.636607 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15\": container with ID starting with 0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15 not found: ID does not exist" containerID="0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15" Oct 30 01:15:20.637038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.636675 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15"} err="failed to get container status \"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15\": rpc error: code = NotFound desc = could not find container \"0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15\": container with ID starting with 0a87afc4e44bbf5afb026fd6d37f63219871aa44e8ece7c7169ad95143341b15 not found: ID does not exist" Oct 30 01:15:20.637038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.636699 5938 scope.go:122] "RemoveContainer" containerID="f7dd08d15ac3dfa69261ea7a31df3fc1caeb8b42e69fde22e4586865adbee814" Oct 30 01:15:20.678928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.678856 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:20.736757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.736730 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:15:20.741811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.741786 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8b2c6-openshift-default-749b97c7b7-vsq6g"] Oct 30 01:15:20.766683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.766659 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\") pod \"e43b551f-5970-4b36-bc8a-ac547d631b86\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " Oct 30 01:15:20.766795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.766725 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\" (UniqueName: \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\") pod \"e43b551f-5970-4b36-bc8a-ac547d631b86\" (UID: \"e43b551f-5970-4b36-bc8a-ac547d631b86\") " Oct 30 01:15:20.767121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.767096 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca" pod "e43b551f-5970-4b36-bc8a-ac547d631b86" (UID: "e43b551f-5970-4b36-bc8a-ac547d631b86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:20.767523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.767506 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b551f-5970-4b36-bc8a-ac547d631b86-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.768933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.768905 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4" pod "e43b551f-5970-4b36-bc8a-ac547d631b86" (UID: "e43b551f-5970-4b36-bc8a-ac547d631b86"). InnerVolumeSpecName "kube-api-access-wbgn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:20.795444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.792959 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:20.800992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.800966 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-6f4fba-29888715-7cwjj"] Oct 30 01:15:20.869369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:20.869341 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgn4\" (UniqueName: \"kubernetes.io/projected/e43b551f-5970-4b36-bc8a-ac547d631b86-kube-api-access-wbgn4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.365028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.364986 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:21.554253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.554204 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:15:21.557534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.557509 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" Oct 30 01:15:21.557684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.557529 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6" event={"ID":"e43b551f-5970-4b36-bc8a-ac547d631b86","Type":"ContainerDied","Data":"14bf9eca2afb49facb0e935048f6cdd6f6cebc173ae4ffd9d92d5b02b1f44fbc"} Oct 30 01:15:21.557684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.557594 5938 scope.go:122] "RemoveContainer" containerID="226b3f69f0d262110ac3ca70fcf09a1a05efe2034d36f413483b8ac08f6ece7c" Oct 30 01:15:21.561007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.560984 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerStarted","Data":"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185"} Oct 30 01:15:21.614391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.614345 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:15:21.621657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.621592 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:21.664377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.664341 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:21.730823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.730790 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:21.733738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.733711 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-93079c-29888715-jjlt6"] Oct 30 01:15:21.738824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.738077 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:21.747979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.747944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:21.852045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.852003 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:21.929729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.928967 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" path="/var/lib/kubelet/pods/34964578-4b4e-4361-aa9b-cc61ce9f9110/volumes" Oct 30 01:15:21.929729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.929157 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:21.929729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.929704 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" path="/var/lib/kubelet/pods/d19a3887-f70f-4dbb-bda6-65c8de4b54fd/volumes" Oct 30 01:15:21.930419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.930395 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" path="/var/lib/kubelet/pods/e43b551f-5970-4b36-bc8a-ac547d631b86/volumes" Oct 30 01:15:22.000826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:21.998391 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:22.070423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.070391 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:15:22.071958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.071931 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:22.072180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.072144 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:22.126888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.121067 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:22.146306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.146016 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\") pod \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " Oct 30 01:15:22.146306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.146219 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\" (UniqueName: \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\") pod \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\" (UID: \"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a\") " Oct 30 01:15:22.148621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.148132 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca" pod "3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" (UID: "3e30fe6a-838f-4021-8cac-1c2b9cf64c2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.150357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.150324 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk" pod "3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" (UID: "3e30fe6a-838f-4021-8cac-1c2b9cf64c2a"). InnerVolumeSpecName "kube-api-access-rlpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.163594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.163512 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:15:22.256359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.256288 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.256359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.256321 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpmk\" (UniqueName: \"kubernetes.io/projected/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a-kube-api-access-rlpmk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.435116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.435083 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:22.442470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.442444 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:22.443644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.443626 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:22.560970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.560900 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\" (UniqueName: \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\") pod \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " Oct 30 01:15:22.560970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.560954 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\") pod \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\" (UID: \"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f\") " Oct 30 01:15:22.561505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561051 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\" (UniqueName: \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\") pod \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " Oct 30 01:15:22.561505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561075 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\") pod \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\" (UID: \"2f0e6954-0e5e-4d32-9ded-2b0b557007a4\") " Oct 30 01:15:22.561505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561114 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\" (UniqueName: \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\") pod \"6e73d49a-bd45-4f0c-a545-77d123952274\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " Oct 30 01:15:22.561505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\") pod \"6e73d49a-bd45-4f0c-a545-77d123952274\" (UID: \"6e73d49a-bd45-4f0c-a545-77d123952274\") " Oct 30 01:15:22.561505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca" pod "b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" (UID: "b2772081-0a7b-4a02-a2f3-ddbb04db7b3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.561799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561647 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca" pod "2f0e6954-0e5e-4d32-9ded-2b0b557007a4" (UID: "2f0e6954-0e5e-4d32-9ded-2b0b557007a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.561799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561752 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca" pod "6e73d49a-bd45-4f0c-a545-77d123952274" (UID: "6e73d49a-bd45-4f0c-a545-77d123952274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:22.561966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561943 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e73d49a-bd45-4f0c-a545-77d123952274-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.562056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561971 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.562056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.561986 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.563586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.563560 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd" pod "2f0e6954-0e5e-4d32-9ded-2b0b557007a4" (UID: "2f0e6954-0e5e-4d32-9ded-2b0b557007a4"). InnerVolumeSpecName "kube-api-access-8wsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.563994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.563973 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7" pod "b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" (UID: "b2772081-0a7b-4a02-a2f3-ddbb04db7b3f"). InnerVolumeSpecName "kube-api-access-v4nv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.564095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.564049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz" pod "6e73d49a-bd45-4f0c-a545-77d123952274" (UID: "6e73d49a-bd45-4f0c-a545-77d123952274"). InnerVolumeSpecName "kube-api-access-n5llz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:22.574855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.574830 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" Oct 30 01:15:22.574996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.574893 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5" event={"ID":"3e30fe6a-838f-4021-8cac-1c2b9cf64c2a","Type":"ContainerDied","Data":"0d1ae160b17c7e1583b99d91de0922e495a64493244a646956d10c882285e1dc"} Oct 30 01:15:22.574996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.574946 5938 scope.go:122] "RemoveContainer" containerID="fbf0894575e11e03cba68d6c22c25b1f0ea1cd5b4b09c5d3b00d4e647322a207" Oct 30 01:15:22.576967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.576624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" event={"ID":"6e73d49a-bd45-4f0c-a545-77d123952274","Type":"ContainerDied","Data":"ab45413f707d989f8f988896426c5ee725c5b93b9cd8e122d13274788d8ddfb9"} Oct 30 01:15:22.576967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.576692 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz" Oct 30 01:15:22.582904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.579050 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" event={"ID":"b2772081-0a7b-4a02-a2f3-ddbb04db7b3f","Type":"ContainerDied","Data":"169b743fe24b4d11f23de07dc27c52c5d4f921785e7516ad64f58f401c836466"} Oct 30 01:15:22.582904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.579146 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh" Oct 30 01:15:22.612045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.583627 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" Oct 30 01:15:22.612045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.593637 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j" event={"ID":"2f0e6954-0e5e-4d32-9ded-2b0b557007a4","Type":"ContainerDied","Data":"4d3d357eeeae0fffbff627a2ed68f4d6d57b74122cce1a68804a9eddb77df68f"} Oct 30 01:15:22.612045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.607195 5938 scope.go:122] "RemoveContainer" containerID="516cd14a840aca2a613057da00aedc8083f15d6cc461bd22944481bee775d6cb" Oct 30 01:15:22.624402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.624378 5938 scope.go:122] "RemoveContainer" containerID="06f1f7c3f7a9bf9b9292d4516a7d3874d2af1a1e066d42d3d7b1a924bdd73803" Oct 30 01:15:22.638403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.638385 5938 scope.go:122] "RemoveContainer" containerID="e9bfdfbba868635418367a1d8d1e28703706d7dd87552616d9d2d4925aa50579" Oct 30 01:15:22.663741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.663716 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5llz\" (UniqueName: \"kubernetes.io/projected/6e73d49a-bd45-4f0c-a545-77d123952274-kube-api-access-n5llz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.663846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.663743 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nv7\" (UniqueName: \"kubernetes.io/projected/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f-kube-api-access-v4nv7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.663846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.663769 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsmd\" (UniqueName: \"kubernetes.io/projected/2f0e6954-0e5e-4d32-9ded-2b0b557007a4-kube-api-access-8wsmd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:22.677904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.677879 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:22.711884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.684677 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-1a11cf-29888715-jnlgh"] Oct 30 01:15:22.738129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.738103 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:22.744407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.744379 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce513-29888715-dzt9j"] Oct 30 01:15:22.774546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.774512 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:22.817588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.817502 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cf4567-29888715-2n9f5"] Oct 30 01:15:22.844974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.844900 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:22.859540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:22.859513 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-swzqz"] Oct 30 01:15:23.068824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.068739 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:23.068824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.068788 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:23.516157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.516126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:23.878756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.878721 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" path="/var/lib/kubelet/pods/2f0e6954-0e5e-4d32-9ded-2b0b557007a4/volumes" Oct 30 01:15:23.879317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.879298 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" path="/var/lib/kubelet/pods/3e30fe6a-838f-4021-8cac-1c2b9cf64c2a/volumes" Oct 30 01:15:23.879744 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.879729 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" path="/var/lib/kubelet/pods/6e73d49a-bd45-4f0c-a545-77d123952274/volumes" Oct 30 01:15:23.880384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:23.880365 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" path="/var/lib/kubelet/pods/b2772081-0a7b-4a02-a2f3-ddbb04db7b3f/volumes" Oct 30 01:15:24.068498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:24.068468 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:24.068667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:24.068517 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:25.085895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:25.068210 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:25.085895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:25.068283 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:25.302887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:25.302830 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:25.854883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:25.854833 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:26.000756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.000669 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:26.068490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.068461 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:26.068660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.068511 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:26.554855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.554810 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" podUID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" containerName="payload-processing" containerID="cri-o://d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840" gracePeriod=25 Oct 30 01:15:26.606384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.606349 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:26.617093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.617058 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" podUID="fbff5826-7c67-4771-ac78-ed79fc8b6636" containerName="payload-pre-processing" containerID="cri-o://b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb" gracePeriod=25 Oct 30 01:15:26.761594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.761559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:26.980022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:26.979996 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:15:27.085379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.085283 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\") pod \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " Oct 30 01:15:27.095163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.095125 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\" (UniqueName: \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\") pod \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\" (UID: \"f61c8c8e-523f-4de1-8e0e-a9456504f36c\") " Oct 30 01:15:27.096778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.088092 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Oct 30 01:15:27.112972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.112931 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Oct 30 01:15:27.113149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.097106 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume" pod "f61c8c8e-523f-4de1-8e0e-a9456504f36c" (UID: "f61c8c8e-523f-4de1-8e0e-a9456504f36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:27.115415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.115384 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv" pod "f61c8c8e-523f-4de1-8e0e-a9456504f36c" (UID: "f61c8c8e-523f-4de1-8e0e-a9456504f36c"). InnerVolumeSpecName "kube-api-access-pkscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:27.203451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.203110 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:15:27.206405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.206369 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61c8c8e-523f-4de1-8e0e-a9456504f36c-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:27.206549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.206409 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkscv\" (UniqueName: \"kubernetes.io/projected/f61c8c8e-523f-4de1-8e0e-a9456504f36c-kube-api-access-pkscv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:27.307902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.307829 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\") pod \"fbff5826-7c67-4771-ac78-ed79fc8b6636\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " Oct 30 01:15:27.308177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.308155 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\" (UniqueName: \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\") pod \"fbff5826-7c67-4771-ac78-ed79fc8b6636\" (UID: \"fbff5826-7c67-4771-ac78-ed79fc8b6636\") " Oct 30 01:15:27.310606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.310581 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx" pod "fbff5826-7c67-4771-ac78-ed79fc8b6636" (UID: "fbff5826-7c67-4771-ac78-ed79fc8b6636"). InnerVolumeSpecName "kube-api-access-jwdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:27.310718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.310660 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume" pod "fbff5826-7c67-4771-ac78-ed79fc8b6636" (UID: "fbff5826-7c67-4771-ac78-ed79fc8b6636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:27.409779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.409743 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdwx\" (UniqueName: \"kubernetes.io/projected/fbff5826-7c67-4771-ac78-ed79fc8b6636-kube-api-access-jwdwx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:27.409779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.409773 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbff5826-7c67-4771-ac78-ed79fc8b6636-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:27.467083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.467028 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:27.683693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.683593 5938 generic.go:356] "Generic (PLEG): container finished" podID="fbff5826-7c67-4771-ac78-ed79fc8b6636" containerID="b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb" exitCode=0 Oct 30 01:15:27.684275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.683695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" event={"ID":"fbff5826-7c67-4771-ac78-ed79fc8b6636","Type":"ContainerDied","Data":"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb"} Oct 30 01:15:27.684275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.683804 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" event={"ID":"fbff5826-7c67-4771-ac78-ed79fc8b6636","Type":"ContainerDied","Data":"358cbc0464d75783d922d7dd06aa36546d9f595ed84c26aa373f7b347f442405"} Oct 30 01:15:27.684275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.683841 5938 scope.go:122] "RemoveContainer" containerID="b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb" Oct 30 01:15:27.684275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.683708 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn" Oct 30 01:15:27.685683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.685660 5938 generic.go:356] "Generic (PLEG): container finished" podID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" containerID="d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840" exitCode=0 Oct 30 01:15:27.685764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.685738 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" event={"ID":"f61c8c8e-523f-4de1-8e0e-a9456504f36c","Type":"ContainerDied","Data":"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840"} Oct 30 01:15:27.685811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.685778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" event={"ID":"f61c8c8e-523f-4de1-8e0e-a9456504f36c","Type":"ContainerDied","Data":"1c65551fda1321239ef99b61a4c6c9f4f5148019ad343e3040bc5a6bb06e479b"} Oct 30 01:15:27.685857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.685844 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k" Oct 30 01:15:27.696661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.696641 5938 scope.go:122] "RemoveContainer" containerID="b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb" Oct 30 01:15:27.697129 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:27.697092 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb\": container with ID starting with b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb not found: ID does not exist" containerID="b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb" Oct 30 01:15:27.697264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.697138 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb"} err="failed to get container status \"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb\": rpc error: code = NotFound desc = could not find container \"b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb\": container with ID starting with b94fa50a22fb59c8c975b798b386243022af3a24d033dc8f238e9f066d99ccbb not found: ID does not exist" Oct 30 01:15:27.697264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.697164 5938 scope.go:122] "RemoveContainer" containerID="d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840" Oct 30 01:15:27.708156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.708136 5938 scope.go:122] "RemoveContainer" containerID="d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840" Oct 30 01:15:27.708471 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:27.708449 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840\": container with ID starting with d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840 not found: ID does not exist" containerID="d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840" Oct 30 01:15:27.708528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.708477 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840"} err="failed to get container status \"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840\": rpc error: code = NotFound desc = could not find container \"d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840\": container with ID starting with d5187391ec502fe4e4bf0aaf6b2ba56a6210313ca773da412f3cf4472e7f8840 not found: ID does not exist" Oct 30 01:15:27.749849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.749823 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:15:27.761455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.761431 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-58f65c44f9-lf7bn"] Oct 30 01:15:27.792533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.792499 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:15:27.795926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.795893 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-9977c47d-9hh6k"] Oct 30 01:15:27.878903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.878857 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" path="/var/lib/kubelet/pods/f61c8c8e-523f-4de1-8e0e-a9456504f36c/volumes" Oct 30 01:15:27.879360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:27.879345 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbff5826-7c67-4771-ac78-ed79fc8b6636" path="/var/lib/kubelet/pods/fbff5826-7c67-4771-ac78-ed79fc8b6636/volumes" Oct 30 01:15:28.074846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:28.074769 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:15:28.076682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:28.076658 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:15:28.836193 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:28.836156 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:15:29.133717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:29.133673 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:30.564501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:30.564452 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:31.179477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.179436 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:31.367906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.367853 5938 scope.go:122] "RemoveContainer" containerID="cccb9561a44836783253d1f7d7059844dea99c7a0cf9e86fa67ba829812e772b" Oct 30 01:15:31.447603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.447577 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:31.456675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.456649 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9"] Oct 30 01:15:31.467987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.467962 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5f387c-f2tb9"] Oct 30 01:15:31.591231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.591193 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:31.883774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:31.883738 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3be788-5ac5-48b2-bde6-df37809bd3ef" path="/var/lib/kubelet/pods/4d3be788-5ac5-48b2-bde6-df37809bd3ef/volumes" Oct 30 01:15:32.011423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:32.011392 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:33.576505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:33.575311 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:35.274252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:35.274214 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:37.180232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:37.180200 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:38.509060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:38.509018 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:38.652881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:38.652840 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:38.787150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:38.787064 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 01:15:38.787325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:38.787276 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" containerID="cri-o://c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa" gracePeriod=30 Oct 30 01:15:38.863813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:38.863783 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:39.010054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:39.010004 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:40.599275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:40.599244 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:40.759078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:40.759047 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:15:40.759235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:40.759107 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:15:40.912783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:40.912754 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.534108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.534073 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.599568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.599538 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:15:42.599746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.599727 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" podUID="ebd06182-e455-455b-bbeb-eb48a512fdd8" containerName="payload-processing" containerID="cri-o://8c63478520a99a57a54e8727255a45610543478d98aca29f67bae7150e92d15a" gracePeriod=30 Oct 30 01:15:42.622184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.622136 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:15:42.622408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.622382 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" podUID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" containerName="payload-pre-processing" containerID="cri-o://ca3cb57f1edc91393107c7b65da44a85253f4f3ff1263d84355bf0f0f4f59064" gracePeriod=30 Oct 30 01:15:42.721722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.721693 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.851245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.849061 5938 generic.go:356] "Generic (PLEG): container finished" podID="ebd06182-e455-455b-bbeb-eb48a512fdd8" containerID="8c63478520a99a57a54e8727255a45610543478d98aca29f67bae7150e92d15a" exitCode=0 Oct 30 01:15:42.851245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.849181 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" event={"ID":"ebd06182-e455-455b-bbeb-eb48a512fdd8","Type":"ContainerDied","Data":"8c63478520a99a57a54e8727255a45610543478d98aca29f67bae7150e92d15a"} Oct 30 01:15:42.851459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.851411 5938 generic.go:356] "Generic (PLEG): container finished" podID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" containerID="ca3cb57f1edc91393107c7b65da44a85253f4f3ff1263d84355bf0f0f4f59064" exitCode=0 Oct 30 01:15:42.851459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.851447 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" event={"ID":"440e8c82-f2a9-4aae-bf7b-c73e79e440a1","Type":"ContainerDied","Data":"ca3cb57f1edc91393107c7b65da44a85253f4f3ff1263d84355bf0f0f4f59064"} Oct 30 01:15:42.866841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:42.866813 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.127533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.127501 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:15:43.136305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.136280 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:15:43.163554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.163527 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.209717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.209686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\") pod \"ebd06182-e455-455b-bbeb-eb48a512fdd8\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " Oct 30 01:15:43.209958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.209937 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\" (UniqueName: \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\") pod \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " Oct 30 01:15:43.210111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210090 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume" pod "ebd06182-e455-455b-bbeb-eb48a512fdd8" (UID: "ebd06182-e455-455b-bbeb-eb48a512fdd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:43.210173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210148 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\" (UniqueName: \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\") pod \"ebd06182-e455-455b-bbeb-eb48a512fdd8\" (UID: \"ebd06182-e455-455b-bbeb-eb48a512fdd8\") " Oct 30 01:15:43.210224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210198 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\") pod \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\" (UID: \"440e8c82-f2a9-4aae-bf7b-c73e79e440a1\") " Oct 30 01:15:43.210656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210625 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume" pod "440e8c82-f2a9-4aae-bf7b-c73e79e440a1" (UID: "440e8c82-f2a9-4aae-bf7b-c73e79e440a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:43.210937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210916 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:43.211055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.210939 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd06182-e455-455b-bbeb-eb48a512fdd8-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:43.212062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.212041 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc" pod "440e8c82-f2a9-4aae-bf7b-c73e79e440a1" (UID: "440e8c82-f2a9-4aae-bf7b-c73e79e440a1"). InnerVolumeSpecName "kube-api-access-m4bzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:43.212194 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.212177 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7" pod "ebd06182-e455-455b-bbeb-eb48a512fdd8" (UID: "ebd06182-e455-455b-bbeb-eb48a512fdd8"). InnerVolumeSpecName "kube-api-access-h5jw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:43.316494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.316453 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.317572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.317543 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bzc\" (UniqueName: \"kubernetes.io/projected/440e8c82-f2a9-4aae-bf7b-c73e79e440a1-kube-api-access-m4bzc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:43.317572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.317572 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jw7\" (UniqueName: \"kubernetes.io/projected/ebd06182-e455-455b-bbeb-eb48a512fdd8-kube-api-access-h5jw7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:43.470154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.470065 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.839118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.839085 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.860560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.860537 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" Oct 30 01:15:43.860662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.860582 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn" event={"ID":"440e8c82-f2a9-4aae-bf7b-c73e79e440a1","Type":"ContainerDied","Data":"f9fe070edd638f5427fc9b93503de230028a90828dc775fd13a7602ae7427cef"} Oct 30 01:15:43.860662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.860650 5938 scope.go:122] "RemoveContainer" containerID="ca3cb57f1edc91393107c7b65da44a85253f4f3ff1263d84355bf0f0f4f59064" Oct 30 01:15:43.862260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.862240 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" event={"ID":"ebd06182-e455-455b-bbeb-eb48a512fdd8","Type":"ContainerDied","Data":"db1327f32953cf3b3cabb228335f5069311d3323ca3b2d976fd5fdd7d752a785"} Oct 30 01:15:43.862347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.862260 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx" Oct 30 01:15:43.900656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.900597 5938 scope.go:122] "RemoveContainer" containerID="8c63478520a99a57a54e8727255a45610543478d98aca29f67bae7150e92d15a" Oct 30 01:15:43.926372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.926329 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:15:43.930280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.930255 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d44c2-7bb4866799-7lhgn"] Oct 30 01:15:43.961794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.961757 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:15:43.965422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:43.965395 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d44c2-76668986b6-947cx"] Oct 30 01:15:44.079674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.079649 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 01:15:44.125982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.125955 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126049 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126090 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126223 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126280 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126306 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:44.126398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126340 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.126398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126426 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126523 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\") pod \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\" (UID: \"618f02c9-7f92-46ee-a3a5-4245d127ca8d\") " Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126715 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.126883 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127092 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:44.127240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127118 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:44.127508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127350 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.127508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127370 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.127508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127383 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.127508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127396 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.127508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.127409 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.129365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.129341 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:44.129463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.129433 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:15:44.129808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.129781 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:15:44.129928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.129916 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w" pod "618f02c9-7f92-46ee-a3a5-4245d127ca8d" (UID: "618f02c9-7f92-46ee-a3a5-4245d127ca8d"). InnerVolumeSpecName "kube-api-access-gxm6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:44.228716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.228685 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.228716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.228712 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.228716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.228721 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618f02c9-7f92-46ee-a3a5-4245d127ca8d-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.228942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.228730 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxm6w\" (UniqueName: \"kubernetes.io/projected/618f02c9-7f92-46ee-a3a5-4245d127ca8d-kube-api-access-gxm6w\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:44.871101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.871065 5938 generic.go:356] "Generic (PLEG): container finished" podID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerID="c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa" exitCode=0 Oct 30 01:15:44.871540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.871130 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" Oct 30 01:15:44.871540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.871156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerDied","Data":"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa"} Oct 30 01:15:44.871540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.871215 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n" event={"ID":"618f02c9-7f92-46ee-a3a5-4245d127ca8d","Type":"ContainerDied","Data":"8a1dad4c4b6563eeed012dee084bf4930f7064f95389c7915aa8a296f3beda91"} Oct 30 01:15:44.871540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.871243 5938 scope.go:122] "RemoveContainer" containerID="c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa" Oct 30 01:15:44.883411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.883389 5938 scope.go:122] "RemoveContainer" containerID="b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a" Oct 30 01:15:44.893449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.893429 5938 scope.go:122] "RemoveContainer" containerID="c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa" Oct 30 01:15:44.893709 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:44.893691 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa\": container with ID starting with c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa not found: ID does not exist" containerID="c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa" Oct 30 01:15:44.893768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.893718 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa"} err="failed to get container status \"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa\": rpc error: code = NotFound desc = could not find container \"c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa\": container with ID starting with c18fb30cf31d15bebe8075d59142283bff9a98172463a5c18b51ddb4331e53fa not found: ID does not exist" Oct 30 01:15:44.893768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.893736 5938 scope.go:122] "RemoveContainer" containerID="b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a" Oct 30 01:15:44.894001 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:44.893985 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a\": container with ID starting with b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a not found: ID does not exist" containerID="b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a" Oct 30 01:15:44.894064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.894004 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a"} err="failed to get container status \"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a\": rpc error: code = NotFound desc = could not find container \"b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a\": container with ID starting with b734428fcdf7ba553a624df06d57b18be84b88825962623bdf7fb7571d172d4a not found: ID does not exist" Oct 30 01:15:44.923308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.923288 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 01:15:44.934165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:44.934146 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5f387c-openshift-default-5ff776bc59-f4q9n"] Oct 30 01:15:45.169814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:45.169743 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:45.878072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:45.878040 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" path="/var/lib/kubelet/pods/440e8c82-f2a9-4aae-bf7b-c73e79e440a1/volumes" Oct 30 01:15:45.878725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:45.878702 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" path="/var/lib/kubelet/pods/618f02c9-7f92-46ee-a3a5-4245d127ca8d/volumes" Oct 30 01:15:45.879327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:45.879308 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd06182-e455-455b-bbeb-eb48a512fdd8" path="/var/lib/kubelet/pods/ebd06182-e455-455b-bbeb-eb48a512fdd8/volumes" Oct 30 01:15:46.973994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:46.973950 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:48.722934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:48.722898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:49.756198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:49.756164 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 01:15:49.785613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:49.785583 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 01:15:49.925266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:49.925238 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:50.072100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:50.072011 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:50.217605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:50.217574 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:50.580462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:50.580408 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:50.882816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:50.882790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.542569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:51.542537 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.695934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:51.695897 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:52.204474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:52.204437 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:53.453397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:53.453367 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:54.757281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.757236 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" podUID="10d38970-bce0-4aff-ba74-fe9a8e23338e" containerName="payload-processing" containerID="cri-o://30a5907c8a65448e1d7602eace65865c7bc5a95c1cdab7ebc5ed00d385c3d7d2" gracePeriod=25 Oct 30 01:15:54.785823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.785790 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" podUID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" containerName="payload-pre-processing" containerID="cri-o://c501150713b53ff6b30e2adb47f15fe61ea1c36bef212e13f66ffddd4e07d6bc" gracePeriod=25 Oct 30 01:15:54.962104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.956608 5938 generic.go:356] "Generic (PLEG): container finished" podID="10d38970-bce0-4aff-ba74-fe9a8e23338e" containerID="30a5907c8a65448e1d7602eace65865c7bc5a95c1cdab7ebc5ed00d385c3d7d2" exitCode=0 Oct 30 01:15:54.962104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.956786 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" event={"ID":"10d38970-bce0-4aff-ba74-fe9a8e23338e","Type":"ContainerDied","Data":"30a5907c8a65448e1d7602eace65865c7bc5a95c1cdab7ebc5ed00d385c3d7d2"} Oct 30 01:15:54.977380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.973548 5938 generic.go:356] "Generic (PLEG): container finished" podID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" containerID="c501150713b53ff6b30e2adb47f15fe61ea1c36bef212e13f66ffddd4e07d6bc" exitCode=0 Oct 30 01:15:54.977380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:54.973651 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" event={"ID":"9dce2374-c3af-4dba-9918-0f87a6d9a5d3","Type":"ContainerDied","Data":"c501150713b53ff6b30e2adb47f15fe61ea1c36bef212e13f66ffddd4e07d6bc"} Oct 30 01:15:55.067668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.067635 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 01:15:55.115572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.115545 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 01:15:55.169994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.169964 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\" (UniqueName: \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\") pod \"10d38970-bce0-4aff-ba74-fe9a8e23338e\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " Oct 30 01:15:55.170642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.170603 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume" pod "10d38970-bce0-4aff-ba74-fe9a8e23338e" (UID: "10d38970-bce0-4aff-ba74-fe9a8e23338e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:55.170957 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.170180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\") pod \"10d38970-bce0-4aff-ba74-fe9a8e23338e\" (UID: \"10d38970-bce0-4aff-ba74-fe9a8e23338e\") " Oct 30 01:15:55.171648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.171624 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10d38970-bce0-4aff-ba74-fe9a8e23338e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:55.172422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.172395 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn" pod "10d38970-bce0-4aff-ba74-fe9a8e23338e" (UID: "10d38970-bce0-4aff-ba74-fe9a8e23338e"). InnerVolumeSpecName "kube-api-access-t9zmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:55.182651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.182614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:55.273050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.273019 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\" (UniqueName: \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\") pod \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " Oct 30 01:15:55.273281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.273255 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\") pod \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\" (UID: \"9dce2374-c3af-4dba-9918-0f87a6d9a5d3\") " Oct 30 01:15:55.273700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.273667 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume" pod "9dce2374-c3af-4dba-9918-0f87a6d9a5d3" (UID: "9dce2374-c3af-4dba-9918-0f87a6d9a5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:55.274223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.274198 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:55.274309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.274231 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zmn\" (UniqueName: \"kubernetes.io/projected/10d38970-bce0-4aff-ba74-fe9a8e23338e-kube-api-access-t9zmn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:55.275449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.275424 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz" pod "9dce2374-c3af-4dba-9918-0f87a6d9a5d3" (UID: "9dce2374-c3af-4dba-9918-0f87a6d9a5d3"). InnerVolumeSpecName "kube-api-access-9l8lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:55.375904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.375879 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8lz\" (UniqueName: \"kubernetes.io/projected/9dce2374-c3af-4dba-9918-0f87a6d9a5d3-kube-api-access-9l8lz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:15:55.982314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.982286 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" Oct 30 01:15:55.982737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.982325 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v" event={"ID":"10d38970-bce0-4aff-ba74-fe9a8e23338e","Type":"ContainerDied","Data":"b2ed316f05d4a468f2bce719aa446e8a9408be4902caa008b360fd224fdd150f"} Oct 30 01:15:55.982737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.982394 5938 scope.go:122] "RemoveContainer" containerID="30a5907c8a65448e1d7602eace65865c7bc5a95c1cdab7ebc5ed00d385c3d7d2" Oct 30 01:15:55.983899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.983883 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" Oct 30 01:15:55.983997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.983882 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr" event={"ID":"9dce2374-c3af-4dba-9918-0f87a6d9a5d3","Type":"ContainerDied","Data":"4a89e4d576bc96481915c4f6232cfc75f13e8c63da7b975248229cc0708365e1"} Oct 30 01:15:55.993005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:55.992978 5938 scope.go:122] "RemoveContainer" containerID="c501150713b53ff6b30e2adb47f15fe61ea1c36bef212e13f66ffddd4e07d6bc" Oct 30 01:15:56.032826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:56.032759 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 01:15:56.044134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:56.044107 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-00e07a-6c7c75ff48-t2w5v"] Oct 30 01:15:56.076639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:56.076616 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 01:15:56.082149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:56.082129 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-00e07a-7f596f48d7-sdcgr"] Oct 30 01:15:56.843991 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:56.843960 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:57.001237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:57.001203 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:57.220349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:57.220321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:57.879549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:57.879512 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d38970-bce0-4aff-ba74-fe9a8e23338e" path="/var/lib/kubelet/pods/10d38970-bce0-4aff-ba74-fe9a8e23338e/volumes" Oct 30 01:15:57.879992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:57.879978 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" path="/var/lib/kubelet/pods/9dce2374-c3af-4dba-9918-0f87a6d9a5d3/volumes" Oct 30 01:15:58.351081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.351008 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.493234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.493199 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.669300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.669235 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 01:15:58.669484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.669462 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" containerID="cri-o://1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1" gracePeriod=30 Oct 30 01:15:58.722156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.722123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.865931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:58.863959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.919596 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:15:58.919504 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:15:59.264301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:15:59.264213 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:00.706378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:00.706343 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:01.426676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:01.426643 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 01:16:01.456825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:01.456789 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 01:16:01.588426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:01.588394 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:01.723029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:01.722918 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:02.439486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:02.439456 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:16:02.439658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:02.439511 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:16:02.538416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:02.538378 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:03.495246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:03.495201 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 01:16:03.496149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:03.496115 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" containerID="cri-o://bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c" gracePeriod=30 Oct 30 01:16:04.046830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.046805 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 01:16:04.128365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.128336 5938 generic.go:356] "Generic (PLEG): container finished" podID="6982266f-1fbe-485d-94bf-a8267348734b" containerID="1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1" exitCode=0 Oct 30 01:16:04.128539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.128398 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" Oct 30 01:16:04.128539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.128420 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerDied","Data":"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1"} Oct 30 01:16:04.128539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.128483 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms" event={"ID":"6982266f-1fbe-485d-94bf-a8267348734b","Type":"ContainerDied","Data":"281556b3c497469afd2873cbb84607366773507d84382cd42b3764aa86faeca1"} Oct 30 01:16:04.128539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.128512 5938 scope.go:122] "RemoveContainer" containerID="1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1" Oct 30 01:16:04.138922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.138897 5938 scope.go:122] "RemoveContainer" containerID="c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3" Oct 30 01:16:04.150215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.150196 5938 scope.go:122] "RemoveContainer" containerID="1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1" Oct 30 01:16:04.150485 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:04.150470 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1\": container with ID starting with 1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1 not found: ID does not exist" containerID="1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1" Oct 30 01:16:04.150537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.150491 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1"} err="failed to get container status \"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1\": rpc error: code = NotFound desc = could not find container \"1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1\": container with ID starting with 1f84f47f755a7e7bd5b1e4a485f265ef505aaa81c3ebf5b54ad2d635b2d83ea1 not found: ID does not exist" Oct 30 01:16:04.150537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.150506 5938 scope.go:122] "RemoveContainer" containerID="c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3" Oct 30 01:16:04.150756 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:04.150739 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3\": container with ID starting with c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3 not found: ID does not exist" containerID="c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3" Oct 30 01:16:04.150801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.150762 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3"} err="failed to get container status \"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3\": rpc error: code = NotFound desc = could not find container \"c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3\": container with ID starting with c8372da3e6b7aa1e7b2576c2846a70cd9e07199aca7eb90f69ec752192c57da3 not found: ID does not exist" Oct 30 01:16:04.222384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222363 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222397 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222461 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222478 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222553 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222583 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222599 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:04.222675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222631 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222697 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222732 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\") pod \"6982266f-1fbe-485d-94bf-a8267348734b\" (UID: \"6982266f-1fbe-485d-94bf-a8267348734b\") " Oct 30 01:16:04.222953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222747 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:04.222953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222899 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:04.223187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.222957 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:04.223348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223327 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:04.223518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223503 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.223594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223519 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.223594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223530 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.223594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223539 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.223594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.223547 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6982266f-1fbe-485d-94bf-a8267348734b-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.224684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.224659 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:04.225175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.225158 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "kube-api-access-c7q2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:04.225278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.225186 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:04.225354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.225337 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo" pod "6982266f-1fbe-485d-94bf-a8267348734b" (UID: "6982266f-1fbe-485d-94bf-a8267348734b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:16:04.325217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.325189 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6982266f-1fbe-485d-94bf-a8267348734b-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.325217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.325215 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.325217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.325226 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7q2g\" (UniqueName: \"kubernetes.io/projected/6982266f-1fbe-485d-94bf-a8267348734b-kube-api-access-c7q2g\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.325491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.325237 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6982266f-1fbe-485d-94bf-a8267348734b-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:04.352707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.352677 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:04.475667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.475630 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 01:16:04.486613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.486587 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-00e07a-openshift-default-56d8b679cf-vf2ms"] Oct 30 01:16:04.557696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:04.557660 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:05.878451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:05.878413 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6982266f-1fbe-485d-94bf-a8267348734b" path="/var/lib/kubelet/pods/6982266f-1fbe-485d-94bf-a8267348734b/volumes" Oct 30 01:16:05.902934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:05.902901 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:06.076901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.076848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.302915 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.303969 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.303987 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304003 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304010 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304024 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4d3be788-5ac5-48b2-bde6-df37809bd3ef" containerName="revoke-keys" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304032 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3be788-5ac5-48b2-bde6-df37809bd3ef" containerName="revoke-keys" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304049 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ebd06182-e455-455b-bbeb-eb48a512fdd8" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304057 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd06182-e455-455b-bbeb-eb48a512fdd8" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304067 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304075 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304086 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304094 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304105 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304112 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304126 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304133 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304143 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304151 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304169 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304177 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304191 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" containerName="payload-pre-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304198 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" containerName="payload-pre-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304214 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304221 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304238 5938 cpu_manager.go:414] "containerMap: removing container" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304246 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304262 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304270 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304279 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304286 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304303 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304310 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304319 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304326 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304342 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304350 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304359 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304367 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304377 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304385 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304394 5938 cpu_manager.go:414] "containerMap: removing container" podUID="10d38970-bce0-4aff-ba74-fe9a8e23338e" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304402 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d38970-bce0-4aff-ba74-fe9a8e23338e" containerName="payload-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304419 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304426 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304437 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304444 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304459 5938 cpu_manager.go:414] "containerMap: removing container" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304466 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304476 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304484 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304497 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304504 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304518 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304525 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304539 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304547 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304555 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304562 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304571 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304578 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304593 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304600 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304610 5938 cpu_manager.go:414] "containerMap: removing container" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304616 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304627 5938 cpu_manager.go:414] "containerMap: removing container" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304635 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304647 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304654 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304683 5938 cpu_manager.go:414] "containerMap: removing container" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304691 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304705 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304712 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304724 5938 cpu_manager.go:414] "containerMap: removing container" podUID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" containerName="payload-pre-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304731 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" containerName="payload-pre-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304744 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304751 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304761 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fbff5826-7c67-4771-ac78-ed79fc8b6636" containerName="payload-pre-processing" Oct 30 01:16:06.306965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.304768 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbff5826-7c67-4771-ac78-ed79fc8b6636" containerName="payload-pre-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326044 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326075 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326087 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326096 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326105 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326114 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326126 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326136 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326144 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a290c9e-086d-4d5b-946a-432864322395" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326158 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326168 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ce73ed-2f86-4c47-b58c-ee6f418560ec" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326179 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326188 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326197 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326206 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326216 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326226 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326241 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="34964578-4b4e-4361-aa9b-cc61ce9f9110" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326251 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326260 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326271 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="440e8c82-f2a9-4aae-bf7b-c73e79e440a1" containerName="payload-pre-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326283 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326293 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326301 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326310 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbff5826-7c67-4771-ac78-ed79fc8b6636" containerName="payload-pre-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326318 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326327 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3be788-5ac5-48b2-bde6-df37809bd3ef" containerName="revoke-keys" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326338 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd06182-e455-455b-bbeb-eb48a512fdd8" containerName="payload-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326346 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326356 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326368 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326377 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dce2374-c3af-4dba-9918-0f87a6d9a5d3" containerName="payload-pre-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326391 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326401 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61c8c8e-523f-4de1-8e0e-a9456504f36c" containerName="payload-processing" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326413 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6982266f-1fbe-485d-94bf-a8267348734b" containerName="istio-proxy" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326424 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.329712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.326432 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d38970-bce0-4aff-ba74-fe9a8e23338e" containerName="payload-processing" Oct 30 01:16:06.365158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.365122 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:16:06.365498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.365478 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.368573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.368535 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqf6\" (UniqueName: \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.368980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.368959 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.408008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.400925 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-01215f-dockercfg-vw4b7\"" Oct 30 01:16:06.408008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.402284 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-01215f\"" Oct 30 01:16:06.429179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.428414 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" containerID="cri-o://54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b" gracePeriod=25 Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.435687 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436848 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436885 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436904 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436913 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436931 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436938 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436962 5938 cpu_manager.go:414] "containerMap: removing container" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436967 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436984 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436989 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.436997 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437002 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437010 5938 cpu_manager.go:414] "containerMap: removing container" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437014 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618f02c9-7f92-46ee-a3a5-4245d127ca8d" containerName="istio-proxy" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437131 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19a3887-f70f-4dbb-bda6-65c8de4b54fd" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437142 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ba60a-1ddd-4523-a4aa-50cdd719390d" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437149 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73d49a-bd45-4f0c-a545-77d123952274" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437157 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57ae6db-ee66-407d-bb6f-81f24f5d2c92" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437167 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0e6954-0e5e-4d32-9ded-2b0b557007a4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437174 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04bcff3-6bb1-47bb-b1ab-0d20743b88e4" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437181 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b551f-5970-4b36-bc8a-ac547d631b86" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437192 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e30fe6a-838f-4021-8cac-1c2b9cf64c2a" containerName="cleanup" Oct 30 01:16:06.437500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.437200 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2772081-0a7b-4a02-a2f3-ddbb04db7b3f" containerName="cleanup" Oct 30 01:16:06.441424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.441404 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.458220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.455663 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:16:06.458220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.457058 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" containerID="cri-o://a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e" gracePeriod=25 Oct 30 01:16:06.469997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.469972 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxxx\" (UniqueName: \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.470179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.470049 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.470251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.470188 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.470382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.470345 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqf6\" (UniqueName: \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.470925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.470905 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.493286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.493252 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqf6\" (UniqueName: \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\") pod \"payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.571898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.571798 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.572114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.572092 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxxx\" (UniqueName: \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.572596 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.572576 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.594744 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.594712 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxxx\" (UniqueName: \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\") pod \"payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.711988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.711952 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:06.765110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.765073 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:06.991449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:06.991419 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:07.032243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.029376 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:16:07.034652 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:07.034616 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3809237_f3dc_47c0_b9e7_441f185b5736.slice/crio-3a2a45ebcd26a1d7467effea801963786cbe63e7e2a77216eee217328a544973 WatchSource:0}: Error finding container 3a2a45ebcd26a1d7467effea801963786cbe63e7e2a77216eee217328a544973: Status 404 returned error can't find the container with id 3a2a45ebcd26a1d7467effea801963786cbe63e7e2a77216eee217328a544973 Oct 30 01:16:07.088966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.088935 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:16:07.089932 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:07.089857 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5444e104_a542_44a4_b5cc_b51537462215.slice/crio-7941377fec95a7c3e8b6222fa148d23598206632817cdee7f2eef039a700e4d9 WatchSource:0}: Error finding container 7941377fec95a7c3e8b6222fa148d23598206632817cdee7f2eef039a700e4d9: Status 404 returned error can't find the container with id 7941377fec95a7c3e8b6222fa148d23598206632817cdee7f2eef039a700e4d9 Oct 30 01:16:07.161467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.161437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" event={"ID":"5444e104-a542-44a4-b5cc-b51537462215","Type":"ContainerStarted","Data":"7941377fec95a7c3e8b6222fa148d23598206632817cdee7f2eef039a700e4d9"} Oct 30 01:16:07.163038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.163017 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" event={"ID":"c3809237-f3dc-47c0-b9e7-441f185b5736","Type":"ContainerStarted","Data":"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38"} Oct 30 01:16:07.163150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.163058 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" event={"ID":"c3809237-f3dc-47c0-b9e7-441f185b5736","Type":"ContainerStarted","Data":"3a2a45ebcd26a1d7467effea801963786cbe63e7e2a77216eee217328a544973"} Oct 30 01:16:07.163235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.163219 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:07.204494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:07.204441 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" podStartSLOduration=1.2044246140000001 podStartE2EDuration="1.204424614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:07.20114044 +0000 UTC m=+2619.441375554" watchObservedRunningTime="2026-10-30 01:16:07.204424614 +0000 UTC m=+2619.444659727" Oct 30 01:16:08.172196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.172163 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" event={"ID":"5444e104-a542-44a4-b5cc-b51537462215","Type":"ContainerStarted","Data":"75b9af66a7611116c39466c0e2d2570fcd041bc04a0571da555dec7d98b401e1"} Oct 30 01:16:08.172642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.172231 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:08.213686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.213640 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" podStartSLOduration=2.21362653 podStartE2EDuration="2.21362653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:08.211031593 +0000 UTC m=+2620.451266706" watchObservedRunningTime="2026-10-30 01:16:08.21362653 +0000 UTC m=+2620.453861643" Oct 30 01:16:08.814035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.810905 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 01:16:08.831151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.831090 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:08.904626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.904584 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.904805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.904658 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.904805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.904793 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.904846 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.904983 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.905033 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.905104 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.905149 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.905267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.905084 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:08.905267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.905191 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\") pod \"bfb92cf4-01e8-4548-b314-115832aa7b20\" (UID: \"bfb92cf4-01e8-4548-b314-115832aa7b20\") " Oct 30 01:16:08.906280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.906251 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb92cf4-01e8-4548-b314-115832aa7b20-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:08.906566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.906540 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:08.906666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.906606 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:08.907031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.907005 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:08.907152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.907133 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:08.908717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.908680 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "kube-api-access-fzx82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:08.909249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.909222 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:08.911019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.910991 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:08.913348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:08.913311 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo" pod "bfb92cf4-01e8-4548-b314-115832aa7b20" (UID: "bfb92cf4-01e8-4548-b314-115832aa7b20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:16:09.007803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007773 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.007803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007801 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.007803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007810 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzx82\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-kube-api-access-fzx82\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.008074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007820 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.008074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007828 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.008074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007836 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.008074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007844 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.008074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.007852 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb92cf4-01e8-4548-b314-115832aa7b20-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:09.182268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.182238 5938 generic.go:356] "Generic (PLEG): container finished" podID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerID="bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c" exitCode=0 Oct 30 01:16:09.182721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.182286 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" Oct 30 01:16:09.182721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.182320 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" event={"ID":"bfb92cf4-01e8-4548-b314-115832aa7b20","Type":"ContainerDied","Data":"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c"} Oct 30 01:16:09.182721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.182382 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487" event={"ID":"bfb92cf4-01e8-4548-b314-115832aa7b20","Type":"ContainerDied","Data":"7f59bf79e029726bdb11331f5eac2d6e1b42fa6fc5e7fb3d1a9bbe00579f44a9"} Oct 30 01:16:09.182721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.182406 5938 scope.go:122] "RemoveContainer" containerID="bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c" Oct 30 01:16:09.194722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.194706 5938 scope.go:122] "RemoveContainer" containerID="bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c" Oct 30 01:16:09.195035 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:09.195009 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c\": container with ID starting with bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c not found: ID does not exist" containerID="bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c" Oct 30 01:16:09.195153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.195041 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c"} err="failed to get container status \"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c\": rpc error: code = NotFound desc = could not find container \"bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c\": container with ID starting with bbadd0c53124474e300af04595ea3522af1605547ed14cd6fad173ce8d9c752c not found: ID does not exist" Oct 30 01:16:09.233784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.233754 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 01:16:09.244594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.244573 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-fv487"] Oct 30 01:16:09.815510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.815472 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 01:16:09.847268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.847244 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 01:16:09.879385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.879358 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" path="/var/lib/kubelet/pods/bfb92cf4-01e8-4548-b314-115832aa7b20/volumes" Oct 30 01:16:09.990743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:09.990701 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:10.139067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:10.139034 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:10.241916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:10.241878 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:10.242282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:10.241968 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:10.578672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:10.578594 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:11.914472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:11.914442 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:12.079951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.079909 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:12.457927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.454507 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:16:12.457927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.455787 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" Oct 30 01:16:12.457927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.455805 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" Oct 30 01:16:12.457927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.456158 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb92cf4-01e8-4548-b314-115832aa7b20" containerName="istio-proxy" Oct 30 01:16:12.482215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.482180 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:16:12.482470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.482456 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:16:12.482732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.482386 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.487741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.487715 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:16:12.487905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.487833 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.509170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.509149 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cf4567-dockercfg-597tw\"" Oct 30 01:16:12.509394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.509376 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cf4567\"" Oct 30 01:16:12.594306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.594277 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.594406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.594395 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfgd\" (UniqueName: \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.594479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.594460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.594589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.594571 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.695313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.695283 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.695446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.695360 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfgd\" (UniqueName: \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.695446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.695390 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.695565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.695459 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.696108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.696089 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.696211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.696185 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.718081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.718022 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfgd\" (UniqueName: \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\") pod \"payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.718214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.718195 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\") pod \"payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:12.800405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.800383 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:12.807536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:12.807521 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:13.053984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:13.053901 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:13.109893 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:13.109843 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cfd741_fd26_4d67_9724_23e47317d156.slice/crio-8ba859a1c90f98c9ee765adaa037425472e26620450b36e6ec1f654940b32222 WatchSource:0}: Error finding container 8ba859a1c90f98c9ee765adaa037425472e26620450b36e6ec1f654940b32222: Status 404 returned error can't find the container with id 8ba859a1c90f98c9ee765adaa037425472e26620450b36e6ec1f654940b32222 Oct 30 01:16:13.110469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:13.110404 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:16:13.122354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:13.122329 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:16:13.125240 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:13.125216 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0612cbf4_a8f8_42fe_bf9c_492b46dd3f9b.slice/crio-75da44bb75c404e5d971c1ba8434b5bff47eeabe29f765b32645a44c26a320e6 WatchSource:0}: Error finding container 75da44bb75c404e5d971c1ba8434b5bff47eeabe29f765b32645a44c26a320e6: Status 404 returned error can't find the container with id 75da44bb75c404e5d971c1ba8434b5bff47eeabe29f765b32645a44c26a320e6 Oct 30 01:16:13.218429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:13.218398 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" event={"ID":"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b","Type":"ContainerStarted","Data":"75da44bb75c404e5d971c1ba8434b5bff47eeabe29f765b32645a44c26a320e6"} Oct 30 01:16:13.219789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:13.219757 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" event={"ID":"a1cfd741-fd26-4d67-9724-23e47317d156","Type":"ContainerStarted","Data":"8ba859a1c90f98c9ee765adaa037425472e26620450b36e6ec1f654940b32222"} Oct 30 01:16:14.228436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.228400 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" event={"ID":"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b","Type":"ContainerStarted","Data":"3200a7287f8f785b8e364c1cfd943382a6b0dc504d2b9334d55e54427b54f053"} Oct 30 01:16:14.228819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.228469 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:14.230142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.230119 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" event={"ID":"a1cfd741-fd26-4d67-9724-23e47317d156","Type":"ContainerStarted","Data":"8821592d4b2fd833253059b98c242c1bb1e8c2d2be616f84224607c52e267b9e"} Oct 30 01:16:14.230229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.230166 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:14.274936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.274892 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" podStartSLOduration=2.274879027 podStartE2EDuration="2.274879027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:14.272596202 +0000 UTC m=+2626.512831317" watchObservedRunningTime="2026-10-30 01:16:14.274879027 +0000 UTC m=+2626.515114132" Oct 30 01:16:14.313950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.313907 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" podStartSLOduration=2.313895033 podStartE2EDuration="2.313895033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:14.3117979 +0000 UTC m=+2626.552033012" watchObservedRunningTime="2026-10-30 01:16:14.313895033 +0000 UTC m=+2626.554130146" Oct 30 01:16:14.805420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.805382 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:14.816105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.816064 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" containerID="cri-o://76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e" gracePeriod=25 Oct 30 01:16:14.847934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:14.847859 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" containerID="cri-o://6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87" gracePeriod=25 Oct 30 01:16:16.417177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:16.417132 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:18.174918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:18.174886 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:16:18.294036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:18.293998 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:18.662585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:18.662554 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:18.662759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:18.662581 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:19.186946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:19.186916 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:16:20.120671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:20.120435 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:20.241620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:20.241576 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:20.242063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:20.241673 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:21.525337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:21.525304 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.124895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:22.124830 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.270215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:22.270174 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.407190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:22.407103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.801074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:22.801002 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:24.172058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:24.171992 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:25.023118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:25.023085 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:25.239733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:25.239705 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:16:25.240231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:25.239786 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:16:25.619462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:25.619422 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.007986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.007896 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.151304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.151259 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.747021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.746988 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 01:16:26.781833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.781814 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 01:16:26.808615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.808591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\" (UniqueName: \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\") pod \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " Oct 30 01:16:26.808854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.808836 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\") pod \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\" (UID: \"83166205-d53c-4d66-8e4f-eb1ec8d92cbb\") " Oct 30 01:16:26.809262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.809237 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume" pod "83166205-d53c-4d66-8e4f-eb1ec8d92cbb" (UID: "83166205-d53c-4d66-8e4f-eb1ec8d92cbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:26.809444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.809426 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:26.810820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.810792 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr" pod "83166205-d53c-4d66-8e4f-eb1ec8d92cbb" (UID: "83166205-d53c-4d66-8e4f-eb1ec8d92cbb"). InnerVolumeSpecName "kube-api-access-lbbdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:26.911477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.911438 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\") pod \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " Oct 30 01:16:26.912340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.912313 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\" (UniqueName: \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\") pod \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\" (UID: \"e3bb250b-c85b-4e1d-88bf-d001d593c29c\") " Oct 30 01:16:26.913003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.912062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume" pod "e3bb250b-c85b-4e1d-88bf-d001d593c29c" (UID: "e3bb250b-c85b-4e1d-88bf-d001d593c29c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:26.913003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.913270 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbdr\" (UniqueName: \"kubernetes.io/projected/83166205-d53c-4d66-8e4f-eb1ec8d92cbb-kube-api-access-lbbdr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:26.913003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.913293 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3bb250b-c85b-4e1d-88bf-d001d593c29c-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:26.916123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:26.915599 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8" pod "e3bb250b-c85b-4e1d-88bf-d001d593c29c" (UID: "e3bb250b-c85b-4e1d-88bf-d001d593c29c"). InnerVolumeSpecName "kube-api-access-lfpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:27.014750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.014692 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpv8\" (UniqueName: \"kubernetes.io/projected/e3bb250b-c85b-4e1d-88bf-d001d593c29c-kube-api-access-lfpv8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:27.348687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.348598 5938 generic.go:356] "Generic (PLEG): container finished" podID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerID="54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b" exitCode=0 Oct 30 01:16:27.348687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.348646 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" Oct 30 01:16:27.349008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.348688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" event={"ID":"83166205-d53c-4d66-8e4f-eb1ec8d92cbb","Type":"ContainerDied","Data":"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b"} Oct 30 01:16:27.349008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.348753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz" event={"ID":"83166205-d53c-4d66-8e4f-eb1ec8d92cbb","Type":"ContainerDied","Data":"3479aaa37c98e80603fee26e8ea963f6a68088ba9786efa9cb147074c98b7b60"} Oct 30 01:16:27.349008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.348785 5938 scope.go:122] "RemoveContainer" containerID="54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b" Oct 30 01:16:27.350276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.350255 5938 generic.go:356] "Generic (PLEG): container finished" podID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerID="a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e" exitCode=0 Oct 30 01:16:27.350406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.350305 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" Oct 30 01:16:27.350406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.350349 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" event={"ID":"e3bb250b-c85b-4e1d-88bf-d001d593c29c","Type":"ContainerDied","Data":"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e"} Oct 30 01:16:27.350523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.350409 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l" event={"ID":"e3bb250b-c85b-4e1d-88bf-d001d593c29c","Type":"ContainerDied","Data":"0462fc2e5d7fc07b8cf81a6b82ee68061afc3d0405a9eef57c15fba052fff65d"} Oct 30 01:16:27.361391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.361376 5938 scope.go:122] "RemoveContainer" containerID="54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b" Oct 30 01:16:27.361641 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:27.361622 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b\": container with ID starting with 54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b not found: ID does not exist" containerID="54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b" Oct 30 01:16:27.361685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.361648 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b"} err="failed to get container status \"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b\": rpc error: code = NotFound desc = could not find container \"54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b\": container with ID starting with 54c38af0921436fa5fc8a811c4867e87626c974934c5e805bd6416a3da386c1b not found: ID does not exist" Oct 30 01:16:27.361685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.361665 5938 scope.go:122] "RemoveContainer" containerID="a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e" Oct 30 01:16:27.371139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.371123 5938 scope.go:122] "RemoveContainer" containerID="a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e" Oct 30 01:16:27.371377 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:27.371359 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e\": container with ID starting with a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e not found: ID does not exist" containerID="a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e" Oct 30 01:16:27.371443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.371383 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e"} err="failed to get container status \"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e\": rpc error: code = NotFound desc = could not find container \"a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e\": container with ID starting with a7dd14dbf4157c01cbb0fb8821ff5632888ba2e8d966c904c0f5d4c21872545e not found: ID does not exist" Oct 30 01:16:27.404025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.403999 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 01:16:27.415592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.415570 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-48df96-5784ccbccf-gdg9l"] Oct 30 01:16:27.439241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.439210 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 01:16:27.445819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.445797 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-48df96-75bb88b5b4-7qhwz"] Oct 30 01:16:27.830225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.830194 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:27.900322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.900290 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" path="/var/lib/kubelet/pods/83166205-d53c-4d66-8e4f-eb1ec8d92cbb/volumes" Oct 30 01:16:27.900817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:27.900799 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" path="/var/lib/kubelet/pods/e3bb250b-c85b-4e1d-88bf-d001d593c29c/volumes" Oct 30 01:16:28.553590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:28.553557 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:28.664554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:28.663466 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:28.664554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:28.664075 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:28.837242 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:28.837161 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:16:29.570644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:29.570581 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:29.726954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:29.726924 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:29.909454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:29.909403 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:30.349253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:30.349161 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:30.551638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:30.551604 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:30.630576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:30.630546 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 01:16:30.630970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:30.630915 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" containerID="cri-o://f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a" gracePeriod=30 Oct 30 01:16:30.712556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:30.712525 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:31.368211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.368181 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:31.509292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.509262 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:16:31.510045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510027 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" Oct 30 01:16:31.510045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510045 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" Oct 30 01:16:31.510167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510071 5938 cpu_manager.go:414] "containerMap: removing container" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" Oct 30 01:16:31.510167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510077 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" Oct 30 01:16:31.510237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510215 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bb250b-c85b-4e1d-88bf-d001d593c29c" containerName="payload-pre-processing" Oct 30 01:16:31.510237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.510234 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="83166205-d53c-4d66-8e4f-eb1ec8d92cbb" containerName="payload-processing" Oct 30 01:16:31.515697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.515679 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.519057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.519036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:31.536484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.536459 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:16:31.544136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.544116 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9a8001\"" Oct 30 01:16:31.544244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.544116 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9a8001-dockercfg-m79xr\"" Oct 30 01:16:31.696888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.691925 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:16:31.715031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.713079 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:16:31.715031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.713216 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.753568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.753531 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.753710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.753692 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.753765 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.753739 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rk5v\" (UniqueName: \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.753801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.753788 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42g5\" (UniqueName: \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.854569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.854544 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.854689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.854639 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.854689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.854665 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rk5v\" (UniqueName: \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.854813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.854694 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42g5\" (UniqueName: \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.855288 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.855264 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.855489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.855466 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:31.877459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.877412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42g5\" (UniqueName: \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:31.877538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:31.877518 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rk5v\" (UniqueName: \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\") pod \"payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:32.048539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.048517 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:32.131541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.131289 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:32.253186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.253144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:32.359249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.359225 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:16:32.361521 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:32.361492 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc33f001_ee60_48d6_b274_7f4569d66dd4.slice/crio-e8f19a66d67146d150df221bee291c1ab72101bf9157faec1659bd76bf54a13f WatchSource:0}: Error finding container e8f19a66d67146d150df221bee291c1ab72101bf9157faec1659bd76bf54a13f: Status 404 returned error can't find the container with id e8f19a66d67146d150df221bee291c1ab72101bf9157faec1659bd76bf54a13f Oct 30 01:16:32.401175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.401135 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" event={"ID":"dc33f001-ee60-48d6-b274-7f4569d66dd4","Type":"ContainerStarted","Data":"e8f19a66d67146d150df221bee291c1ab72101bf9157faec1659bd76bf54a13f"} Oct 30 01:16:32.453452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:32.453429 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:16:32.454681 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:32.454654 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7831abe_18da_43bc_81d7_700c42b2a306.slice/crio-787eeda20e80e00538e078b01dde2e9f59e27a2e09cc733e7370e1591024f321 WatchSource:0}: Error finding container 787eeda20e80e00538e078b01dde2e9f59e27a2e09cc733e7370e1591024f321: Status 404 returned error can't find the container with id 787eeda20e80e00538e078b01dde2e9f59e27a2e09cc733e7370e1591024f321 Oct 30 01:16:33.410741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.410700 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" event={"ID":"dc33f001-ee60-48d6-b274-7f4569d66dd4","Type":"ContainerStarted","Data":"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7"} Oct 30 01:16:33.411212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.410773 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:33.412465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.412443 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" event={"ID":"a7831abe-18da-43bc-81d7-700c42b2a306","Type":"ContainerStarted","Data":"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c"} Oct 30 01:16:33.412536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.412487 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" event={"ID":"a7831abe-18da-43bc-81d7-700c42b2a306","Type":"ContainerStarted","Data":"787eeda20e80e00538e078b01dde2e9f59e27a2e09cc733e7370e1591024f321"} Oct 30 01:16:33.412578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.412541 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:33.452229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.452185 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" podStartSLOduration=2.452170148 podStartE2EDuration="2.452170148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:33.449500064 +0000 UTC m=+2645.689735180" watchObservedRunningTime="2026-10-30 01:16:33.452170148 +0000 UTC m=+2645.692405265" Oct 30 01:16:33.488726 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:33.488669 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" podStartSLOduration=2.48865338 podStartE2EDuration="2.48865338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:33.487757502 +0000 UTC m=+2645.727992613" watchObservedRunningTime="2026-10-30 01:16:33.48865338 +0000 UTC m=+2645.728888494" Oct 30 01:16:34.111673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:34.111639 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:35.141278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.141253 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 01:16:35.178699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.178677 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 01:16:35.195574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.195534 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\") pod \"489d8d84-065a-455b-aba1-298d85025ca7\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " Oct 30 01:16:35.195711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.195686 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\" (UniqueName: \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\") pod \"7482d7ef-6951-40d2-b581-39b19c0d35a2\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " Oct 30 01:16:35.195799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.195786 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\") pod \"7482d7ef-6951-40d2-b581-39b19c0d35a2\" (UID: \"7482d7ef-6951-40d2-b581-39b19c0d35a2\") " Oct 30 01:16:35.195917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.195902 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\" (UniqueName: \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\") pod \"489d8d84-065a-455b-aba1-298d85025ca7\" (UID: \"489d8d84-065a-455b-aba1-298d85025ca7\") " Oct 30 01:16:35.196020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.195992 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume" pod "489d8d84-065a-455b-aba1-298d85025ca7" (UID: "489d8d84-065a-455b-aba1-298d85025ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:35.196396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.196376 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489d8d84-065a-455b-aba1-298d85025ca7-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:35.196396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.196381 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume" pod "7482d7ef-6951-40d2-b581-39b19c0d35a2" (UID: "7482d7ef-6951-40d2-b581-39b19c0d35a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:35.198348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.198316 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658" pod "489d8d84-065a-455b-aba1-298d85025ca7" (UID: "489d8d84-065a-455b-aba1-298d85025ca7"). InnerVolumeSpecName "kube-api-access-q4658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:35.198733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.198715 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h" pod "7482d7ef-6951-40d2-b581-39b19c0d35a2" (UID: "7482d7ef-6951-40d2-b581-39b19c0d35a2"). InnerVolumeSpecName "kube-api-access-bxn6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:35.297543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.297510 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn6h\" (UniqueName: \"kubernetes.io/projected/7482d7ef-6951-40d2-b581-39b19c0d35a2-kube-api-access-bxn6h\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:35.297543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.297544 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7482d7ef-6951-40d2-b581-39b19c0d35a2-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:35.297822 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.297559 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4658\" (UniqueName: \"kubernetes.io/projected/489d8d84-065a-455b-aba1-298d85025ca7-kube-api-access-q4658\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:35.429117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.429049 5938 generic.go:356] "Generic (PLEG): container finished" podID="489d8d84-065a-455b-aba1-298d85025ca7" containerID="6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87" exitCode=0 Oct 30 01:16:35.429117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.429093 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" Oct 30 01:16:35.429319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.429125 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" event={"ID":"489d8d84-065a-455b-aba1-298d85025ca7","Type":"ContainerDied","Data":"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87"} Oct 30 01:16:35.429319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.429204 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm" event={"ID":"489d8d84-065a-455b-aba1-298d85025ca7","Type":"ContainerDied","Data":"46f50304e666d4264931c764f70d52677a4ed9db2e1612943863ffb68155d494"} Oct 30 01:16:35.429319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.429236 5938 scope.go:122] "RemoveContainer" containerID="6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87" Oct 30 01:16:35.430688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.430664 5938 generic.go:356] "Generic (PLEG): container finished" podID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerID="76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e" exitCode=0 Oct 30 01:16:35.430802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.430706 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" event={"ID":"7482d7ef-6951-40d2-b581-39b19c0d35a2","Type":"ContainerDied","Data":"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e"} Oct 30 01:16:35.430802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.430715 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" Oct 30 01:16:35.430933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.430763 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx" event={"ID":"7482d7ef-6951-40d2-b581-39b19c0d35a2","Type":"ContainerDied","Data":"5f612468b0a16eeb947243cb89c4e27866c988bdac35220cfd5a4997d4f80f53"} Oct 30 01:16:35.445434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.445415 5938 scope.go:122] "RemoveContainer" containerID="6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87" Oct 30 01:16:35.445702 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:35.445683 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87\": container with ID starting with 6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87 not found: ID does not exist" containerID="6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87" Oct 30 01:16:35.445784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.445711 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87"} err="failed to get container status \"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87\": rpc error: code = NotFound desc = could not find container \"6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87\": container with ID starting with 6528ac296285524ef1757f25c91e2bcf0deea12ab3793776d8f87b7aafae3e87 not found: ID does not exist" Oct 30 01:16:35.445784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.445734 5938 scope.go:122] "RemoveContainer" containerID="76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e" Oct 30 01:16:35.455334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.455310 5938 scope.go:122] "RemoveContainer" containerID="76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e" Oct 30 01:16:35.455600 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:35.455580 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e\": container with ID starting with 76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e not found: ID does not exist" containerID="76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e" Oct 30 01:16:35.455664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.455606 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e"} err="failed to get container status \"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e\": rpc error: code = NotFound desc = could not find container \"76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e\": container with ID starting with 76a60c6ffd05b09df8ee7fd3a10e8dde25b824739f915288f197aee22ab7dd5e not found: ID does not exist" Oct 30 01:16:35.480403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.480371 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 01:16:35.483768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.483738 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-52650f-5768bb8d4b-r95hx"] Oct 30 01:16:35.515493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.515442 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 01:16:35.519091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.519072 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-52650f-7686c45df9-r8bkm"] Oct 30 01:16:35.884055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.884000 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489d8d84-065a-455b-aba1-298d85025ca7" path="/var/lib/kubelet/pods/489d8d84-065a-455b-aba1-298d85025ca7/volumes" Oct 30 01:16:35.884830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.884743 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" path="/var/lib/kubelet/pods/7482d7ef-6951-40d2-b581-39b19c0d35a2/volumes" Oct 30 01:16:35.920962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:35.920944 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 01:16:36.006753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.006675 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.006921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.006784 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.006921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.006879 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.006932 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.006991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007150 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007184 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007238 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:36.007283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007264 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007321 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\") pod \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\" (UID: \"5a7f95f3-705b-480c-8c51-f9f24766b9f8\") " Oct 30 01:16:36.007447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007394 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:36.007529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:36.007960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.007939 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:16:36.008090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.008075 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.008166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.008091 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.008166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.008101 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.008166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.008112 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.008400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.008377 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:36.009831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.009805 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "kube-api-access-cnpxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:36.010831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.010515 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:36.010831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.010714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:16:36.011338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.011296 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token" pod "5a7f95f3-705b-480c-8c51-f9f24766b9f8" (UID: "5a7f95f3-705b-480c-8c51-f9f24766b9f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:36.109855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.109819 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.109855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.109849 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.110069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.109900 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.110069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.109913 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpxz\" (UniqueName: \"kubernetes.io/projected/5a7f95f3-705b-480c-8c51-f9f24766b9f8-kube-api-access-cnpxz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.110069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.109927 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a7f95f3-705b-480c-8c51-f9f24766b9f8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:36.372956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.372892 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:36.440247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.440218 5938 generic.go:356] "Generic (PLEG): container finished" podID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerID="f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a" exitCode=0 Oct 30 01:16:36.440439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.440271 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" Oct 30 01:16:36.440439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.440305 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerDied","Data":"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a"} Oct 30 01:16:36.440439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.440359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk" event={"ID":"5a7f95f3-705b-480c-8c51-f9f24766b9f8","Type":"ContainerDied","Data":"38e7117a7c139df03f5cbe7673b9ea98ea1e1affb2acd878c2e2cd7193f8dad2"} Oct 30 01:16:36.440439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.440391 5938 scope.go:122] "RemoveContainer" containerID="f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a" Oct 30 01:16:36.453651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.453633 5938 scope.go:122] "RemoveContainer" containerID="69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83" Oct 30 01:16:36.463172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.463155 5938 scope.go:122] "RemoveContainer" containerID="f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a" Oct 30 01:16:36.463431 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:36.463411 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a\": container with ID starting with f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a not found: ID does not exist" containerID="f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a" Oct 30 01:16:36.463516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.463441 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a"} err="failed to get container status \"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a\": rpc error: code = NotFound desc = could not find container \"f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a\": container with ID starting with f0e810f1f0102dbd5a44e84682b58cafaab8f1be7849acb8a623ca87ced4101a not found: ID does not exist" Oct 30 01:16:36.463516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.463462 5938 scope.go:122] "RemoveContainer" containerID="69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83" Oct 30 01:16:36.463726 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:36.463708 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83\": container with ID starting with 69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83 not found: ID does not exist" containerID="69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83" Oct 30 01:16:36.463783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.463732 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83"} err="failed to get container status \"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83\": rpc error: code = NotFound desc = could not find container \"69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83\": container with ID starting with 69edda50566c1ed7e0b08b5c3550e05da0fb2831f2eb13026f25376ac72b8e83 not found: ID does not exist" Oct 30 01:16:36.491591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.491569 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 01:16:36.503186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:36.503159 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-5rlnk"] Oct 30 01:16:37.879668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:37.879632 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" path="/var/lib/kubelet/pods/5a7f95f3-705b-480c-8c51-f9f24766b9f8/volumes" Oct 30 01:16:37.930178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:37.930150 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:39.683795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:39.683759 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:40.718872 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:40.718838 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:41.394470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:41.394432 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:41.804202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:41.804105 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:41.851173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:41.851130 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 01:16:41.883469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:41.883438 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 01:16:41.951946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:41.951914 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:42.105941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:42.105908 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:42.674467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:42.674430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:43.111709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:43.111676 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:16:43.193742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:43.193696 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:43.582739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:43.582653 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:43.831764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:43.831722 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:44.334426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.334394 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:16:44.335669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335639 5938 cpu_manager.go:414] "containerMap: removing container" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" Oct 30 01:16:44.335669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335669 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335689 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335696 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335706 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335716 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335730 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.335887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335737 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.336214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335955 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="489d8d84-065a-455b-aba1-298d85025ca7" containerName="payload-pre-processing" Oct 30 01:16:44.336214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335975 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7482d7ef-6951-40d2-b581-39b19c0d35a2" containerName="payload-processing" Oct 30 01:16:44.336214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335984 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.336214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.335994 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7f95f3-705b-480c-8c51-f9f24766b9f8" containerName="istio-proxy" Oct 30 01:16:44.341763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.341741 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.358293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.358252 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-812ba2-openshift-default-dockercfg-bdlxq\"" Oct 30 01:16:44.364603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.364574 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:16:44.422856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.422834 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:16:44.423006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.422934 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:16:44.436160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.436136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:44.475985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.475958 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.475997 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476145 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzgf\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476219 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476313 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476352 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476430 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476469 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.476949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.476923 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578358 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578400 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578446 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578478 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578535 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578739 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578776 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578822 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.578950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.578905 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzgf\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.579074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.579003 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.579303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.579276 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.579433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.579366 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.579492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.579430 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.579538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.579512 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.580710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.580693 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.581001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.580984 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.600496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.600444 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.600996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.600978 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzgf\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\") pod \"e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:44.606788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.606766 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:44.672886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:44.665158 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:45.084182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.084151 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:16:45.085562 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:45.085534 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8899fba_c9b1_4d54_9f33_5708dd58b119.slice/crio-8ab5eb599b93435f2e49c496cc52890c6ef5d83e49489dba4ffee7556a60ef0a WatchSource:0}: Error finding container 8ab5eb599b93435f2e49c496cc52890c6ef5d83e49489dba4ffee7556a60ef0a: Status 404 returned error can't find the container with id 8ab5eb599b93435f2e49c496cc52890c6ef5d83e49489dba4ffee7556a60ef0a Oct 30 01:16:45.088399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.088361 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:45.088492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.088429 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:45.088492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.088459 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:16:45.568000 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.567961 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerStarted","Data":"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228"} Oct 30 01:16:45.568547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.568022 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerStarted","Data":"8ab5eb599b93435f2e49c496cc52890c6ef5d83e49489dba4ffee7556a60ef0a"} Oct 30 01:16:45.581544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.581519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:45.614724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.613931 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podStartSLOduration=1.613859165 podStartE2EDuration="1.613859165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:45.61079085 +0000 UTC m=+2657.851025965" watchObservedRunningTime="2026-10-30 01:16:45.613859165 +0000 UTC m=+2657.854094303" Oct 30 01:16:45.670613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.666804 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:45.724913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:45.724880 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:46.672530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:46.672500 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:46.852839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:46.852794 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" containerID="cri-o://4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75" gracePeriod=25 Oct 30 01:16:46.884736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:46.884688 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" containerID="cri-o://1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d" gracePeriod=25 Oct 30 01:16:47.415060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:47.415027 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:47.582679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:47.582646 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:47.583547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:47.583527 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:16:48.064315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:48.064278 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:48.231440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:48.231408 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:48.374746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:48.374709 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:48.511376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:48.511340 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:48.665477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:48.665393 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:49.025755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:49.025675 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:49.901524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:49.901494 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:50.168428 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.168352 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:50.325364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.325329 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:50.434076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.433990 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:16:50.487924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.484782 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.487924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.485790 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:16:50.506062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.506013 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812ba2\"" Oct 30 01:16:50.517633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.517606 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812ba2-dockercfg-bjhm6\"" Oct 30 01:16:50.628302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.628266 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.628470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.628407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhcl\" (UniqueName: \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.729558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.729486 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhcl\" (UniqueName: \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.729684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.729656 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.730418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.730397 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.752251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.752225 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhcl\" (UniqueName: \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:50.822569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:50.822544 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:51.124215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.124182 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:16:51.125443 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:16:51.125413 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ae5323_5bfc_49ae_bf2f_9b9edea05fc8.slice/crio-9ec3bc01c8f9c2b4e6b1d46a59361bfa403ac2758f88625e0a1444d2bc8f31f9 WatchSource:0}: Error finding container 9ec3bc01c8f9c2b4e6b1d46a59361bfa403ac2758f88625e0a1444d2bc8f31f9: Status 404 returned error can't find the container with id 9ec3bc01c8f9c2b4e6b1d46a59361bfa403ac2758f88625e0a1444d2bc8f31f9 Oct 30 01:16:51.614954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.614919 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" event={"ID":"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8","Type":"ContainerStarted","Data":"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206"} Oct 30 01:16:51.615121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.614974 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" event={"ID":"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8","Type":"ContainerStarted","Data":"9ec3bc01c8f9c2b4e6b1d46a59361bfa403ac2758f88625e0a1444d2bc8f31f9"} Oct 30 01:16:51.615121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.615009 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:16:51.657854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.657797 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" podStartSLOduration=1.65777922 podStartE2EDuration="1.65777922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.655837965 +0000 UTC m=+2663.896073076" watchObservedRunningTime="2026-10-30 01:16:51.65777922 +0000 UTC m=+2663.898014335" Oct 30 01:16:51.721039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:51.721006 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:53.769283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:53.769228 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:54.769007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:54.768972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:55.095028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:55.094951 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:55.236224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:55.236188 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:55.860068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:55.860034 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:55.860242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:55.860097 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:16:56.189973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:56.189894 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:57.860941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:57.860907 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:58.135309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.135230 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerName="main" containerID="cri-o://daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751" gracePeriod=45 Oct 30 01:16:58.421952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.421928 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:16:58.553095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553065 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553177 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553207 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553314 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:58.553375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553359 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553399 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\" (UniqueName: \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553444 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:58.553601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553495 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\") pod \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\" (UID: \"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3\") " Oct 30 01:16:58.553601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.553527 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:58.554071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.554046 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.554071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.554070 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.554309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.554079 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.555447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.555420 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:58.555551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.555493 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "kube-api-access-mtt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:58.555551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.555494 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs" pod "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" (UID: "5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:58.655393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.655338 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.655393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.655363 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtt7s\" (UniqueName: \"kubernetes.io/projected/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-kube-api-access-mtt7s\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.655393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.655375 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:16:58.795512 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.795484 5938 generic.go:356] "Generic (PLEG): container finished" podID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerID="daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751" exitCode=0 Oct 30 01:16:58.795637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.795521 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" Oct 30 01:16:58.795637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.795554 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" event={"ID":"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3","Type":"ContainerDied","Data":"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751"} Oct 30 01:16:58.795637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.795595 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" event={"ID":"5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3","Type":"ContainerDied","Data":"f4f47346f5cc6a8b5eaf980a068975627da74c15fa5e57cf8f3b4b4bb5a935f3"} Oct 30 01:16:58.795637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.795613 5938 scope.go:122] "RemoveContainer" containerID="daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751" Oct 30 01:16:58.807173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.807155 5938 scope.go:122] "RemoveContainer" containerID="daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751" Oct 30 01:16:58.807423 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:58.807404 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751\": container with ID starting with daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751 not found: ID does not exist" containerID="daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751" Oct 30 01:16:58.807507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.807430 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751"} err="failed to get container status \"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751\": rpc error: code = NotFound desc = could not find container \"daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751\": container with ID starting with daa7f6fbfc4010d101dfc80576f4676c3ab726f893e58aba702f563e4957c751 not found: ID does not exist" Oct 30 01:16:58.838806 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:16:58.838770 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:16:58.846838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.846809 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:16:58.858168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:58.858144 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f"] Oct 30 01:16:59.397592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:59.397552 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-a3619c/auth-test-model-a3619c-kserve-59cc76d697-pwz2f" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerName="main" probeResult="failure" output="Get \"https://10.134.1.18:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:16:59.519286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:59.519257 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.877456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:16:59.877422 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" path="/var/lib/kubelet/pods/5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3/volumes" Oct 30 01:17:01.126693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:01.126654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:02.628673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:02.628641 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:17:02.804881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:02.804832 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:04.488856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:04.488826 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:05.860142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:05.860099 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:05.860538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:05.860198 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:06.210021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:06.209936 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:07.168725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.168700 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 01:17:07.195025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.195001 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 01:17:07.262791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.262762 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\") pod \"8fe31be1-f392-4843-9ec7-6602fb106b53\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " Oct 30 01:17:07.262990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.262856 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\" (UniqueName: \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\") pod \"8fe31be1-f392-4843-9ec7-6602fb106b53\" (UID: \"8fe31be1-f392-4843-9ec7-6602fb106b53\") " Oct 30 01:17:07.263195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.263174 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume" pod "8fe31be1-f392-4843-9ec7-6602fb106b53" (UID: "8fe31be1-f392-4843-9ec7-6602fb106b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:07.263476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.263459 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fe31be1-f392-4843-9ec7-6602fb106b53-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:07.265074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.265048 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87" pod "8fe31be1-f392-4843-9ec7-6602fb106b53" (UID: "8fe31be1-f392-4843-9ec7-6602fb106b53"). InnerVolumeSpecName "kube-api-access-jxw87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:07.363942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.363911 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\") pod \"6128a092-1455-4238-b4d6-57966665f1e3\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " Oct 30 01:17:07.364153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.364136 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\" (UniqueName: \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\") pod \"6128a092-1455-4238-b4d6-57966665f1e3\" (UID: \"6128a092-1455-4238-b4d6-57966665f1e3\") " Oct 30 01:17:07.364291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.364264 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume" pod "6128a092-1455-4238-b4d6-57966665f1e3" (UID: "6128a092-1455-4238-b4d6-57966665f1e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:07.364604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.364588 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxw87\" (UniqueName: \"kubernetes.io/projected/8fe31be1-f392-4843-9ec7-6602fb106b53-kube-api-access-jxw87\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:07.364688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.364609 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6128a092-1455-4238-b4d6-57966665f1e3-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:07.366127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.366102 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd" pod "6128a092-1455-4238-b4d6-57966665f1e3" (UID: "6128a092-1455-4238-b4d6-57966665f1e3"). InnerVolumeSpecName "kube-api-access-tmmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:07.465418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.465362 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmfd\" (UniqueName: \"kubernetes.io/projected/6128a092-1455-4238-b4d6-57966665f1e3-kube-api-access-tmmfd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:07.841424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.841337 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:07.888140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.888099 5938 generic.go:356] "Generic (PLEG): container finished" podID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerID="4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75" exitCode=0 Oct 30 01:17:07.888301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.888215 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" event={"ID":"8fe31be1-f392-4843-9ec7-6602fb106b53","Type":"ContainerDied","Data":"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75"} Oct 30 01:17:07.888301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.888256 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" event={"ID":"8fe31be1-f392-4843-9ec7-6602fb106b53","Type":"ContainerDied","Data":"5f3fbe1f6da28275b3de659e6fc243272c89f09e04eacfd9f8c1f5199ce10679"} Oct 30 01:17:07.888301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.888285 5938 scope.go:122] "RemoveContainer" containerID="4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75" Oct 30 01:17:07.888491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.888420 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb" Oct 30 01:17:07.893121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.893091 5938 generic.go:356] "Generic (PLEG): container finished" podID="6128a092-1455-4238-b4d6-57966665f1e3" containerID="1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d" exitCode=0 Oct 30 01:17:07.893252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.893121 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" Oct 30 01:17:07.893252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.893167 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" event={"ID":"6128a092-1455-4238-b4d6-57966665f1e3","Type":"ContainerDied","Data":"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d"} Oct 30 01:17:07.893252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.893210 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv" event={"ID":"6128a092-1455-4238-b4d6-57966665f1e3","Type":"ContainerDied","Data":"a05dc89ce92447245565e11ccb26d1c4fb8b2aca6303c7d8306d5f159ea54fc3"} Oct 30 01:17:07.905922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.905905 5938 scope.go:122] "RemoveContainer" containerID="4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75" Oct 30 01:17:07.906189 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:07.906169 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75\": container with ID starting with 4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75 not found: ID does not exist" containerID="4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75" Oct 30 01:17:07.906264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.906196 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75"} err="failed to get container status \"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75\": rpc error: code = NotFound desc = could not find container \"4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75\": container with ID starting with 4e12773d408ef0923bdedb8f1889ea6789df18d561a6babf900c34435c394c75 not found: ID does not exist" Oct 30 01:17:07.906264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.906212 5938 scope.go:122] "RemoveContainer" containerID="1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d" Oct 30 01:17:07.915513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.915482 5938 scope.go:122] "RemoveContainer" containerID="1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d" Oct 30 01:17:07.915800 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:07.915777 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d\": container with ID starting with 1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d not found: ID does not exist" containerID="1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d" Oct 30 01:17:07.915940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.915805 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d"} err="failed to get container status \"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d\": rpc error: code = NotFound desc = could not find container \"1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d\": container with ID starting with 1a5bb0fa43c97d9974a8f12161d74ab586d1d2d612be01e4a1d8ea27e4e4e48d not found: ID does not exist" Oct 30 01:17:07.941448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.941424 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 01:17:07.952434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.952410 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-cce513-7c484dd7d8-dl5vb"] Oct 30 01:17:07.981025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.980994 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 01:17:07.984063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:07.984044 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-cce513-6f5d768c59-dnpdv"] Oct 30 01:17:09.695819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:09.695782 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:09.882551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:09.882511 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6128a092-1455-4238-b4d6-57966665f1e3" path="/var/lib/kubelet/pods/6128a092-1455-4238-b4d6-57966665f1e3/volumes" Oct 30 01:17:09.883129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:09.883105 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" path="/var/lib/kubelet/pods/8fe31be1-f392-4843-9ec7-6602fb106b53/volumes" Oct 30 01:17:11.450185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:11.450154 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.604398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:11.604362 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.746043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:11.745910 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.332858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:13.332818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.203988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:15.203944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.923525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:16.923491 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:19.097578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:19.097539 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:19.502368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:19.502328 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:20.848647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:20.848614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:22.221367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.221335 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 01:17:22.221901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.221570 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" podUID="ea0aaf4e-f905-4fd1-9276-1389c2261810" containerName="maas-api" containerID="cri-o://e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9" gracePeriod=30 Oct 30 01:17:22.323074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.323046 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:22.550297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.550257 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 01:17:22.677068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.677034 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\") pod \"ea0aaf4e-f905-4fd1-9276-1389c2261810\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " Oct 30 01:17:22.677284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.677144 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\" (UniqueName: \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\") pod \"ea0aaf4e-f905-4fd1-9276-1389c2261810\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " Oct 30 01:17:22.677356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.677316 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\") pod \"ea0aaf4e-f905-4fd1-9276-1389c2261810\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " Oct 30 01:17:22.677464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.677449 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\") pod \"ea0aaf4e-f905-4fd1-9276-1389c2261810\" (UID: \"ea0aaf4e-f905-4fd1-9276-1389c2261810\") " Oct 30 01:17:22.677786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.677756 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca" pod "ea0aaf4e-f905-4fd1-9276-1389c2261810" (UID: "ea0aaf4e-f905-4fd1-9276-1389c2261810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:22.678343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.678307 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0aaf4e-f905-4fd1-9276-1389c2261810-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:22.679665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.679635 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f" pod "ea0aaf4e-f905-4fd1-9276-1389c2261810" (UID: "ea0aaf4e-f905-4fd1-9276-1389c2261810"). InnerVolumeSpecName "kube-api-access-6tl9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:22.679839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.679819 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls" pod "ea0aaf4e-f905-4fd1-9276-1389c2261810" (UID: "ea0aaf4e-f905-4fd1-9276-1389c2261810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:22.680374 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.680344 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs" pod "ea0aaf4e-f905-4fd1-9276-1389c2261810" (UID: "ea0aaf4e-f905-4fd1-9276-1389c2261810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:22.779967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.779875 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:22.789962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.789927 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tl9f\" (UniqueName: \"kubernetes.io/projected/ea0aaf4e-f905-4fd1-9276-1389c2261810-kube-api-access-6tl9f\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:22.790219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.790199 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0aaf4e-f905-4fd1-9276-1389c2261810-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:22.830787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.830757 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:22.988948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:22.988227 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:23.019183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.019148 5938 generic.go:356] "Generic (PLEG): container finished" podID="ea0aaf4e-f905-4fd1-9276-1389c2261810" containerID="e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9" exitCode=0 Oct 30 01:17:23.019357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.019199 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" Oct 30 01:17:23.019357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.019228 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" event={"ID":"ea0aaf4e-f905-4fd1-9276-1389c2261810","Type":"ContainerDied","Data":"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9"} Oct 30 01:17:23.019357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.019289 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj" event={"ID":"ea0aaf4e-f905-4fd1-9276-1389c2261810","Type":"ContainerDied","Data":"6be37547e611bf6cf36c8226f46bb9d30193ee56292d3690302b35d8f979f237"} Oct 30 01:17:23.019357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.019315 5938 scope.go:122] "RemoveContainer" containerID="e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9" Oct 30 01:17:23.030371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.030353 5938 scope.go:122] "RemoveContainer" containerID="e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9" Oct 30 01:17:23.030632 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:23.030614 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9\": container with ID starting with e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9 not found: ID does not exist" containerID="e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9" Oct 30 01:17:23.030680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.030641 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9"} err="failed to get container status \"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9\": rpc error: code = NotFound desc = could not find container \"e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9\": container with ID starting with e0ca234f1e86a1d960fc86ebb95a0cc077f903f587e4c2c5e8f0e46ece9f67b9 not found: ID does not exist" Oct 30 01:17:23.070011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.069987 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 01:17:23.073978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.073958 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-cce513-6df69b4ccc-lvktj"] Oct 30 01:17:23.134956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.134927 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:23.524000 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.523962 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:23.878417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:23.878385 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0aaf4e-f905-4fd1-9276-1389c2261810" path="/var/lib/kubelet/pods/ea0aaf4e-f905-4fd1-9276-1389c2261810/volumes" Oct 30 01:17:24.597381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.597343 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:24.769943 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.769903 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.994898 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.995981 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996001 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996030 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996037 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996064 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ea0aaf4e-f905-4fd1-9276-1389c2261810" containerName="maas-api" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996072 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0aaf4e-f905-4fd1-9276-1389c2261810" containerName="maas-api" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996098 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerName="main" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996107 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerName="main" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996287 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0aaf4e-f905-4fd1-9276-1389c2261810" containerName="maas-api" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996303 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df8c94c-0423-4a9a-aba7-e7b0a55eb6b3" containerName="main" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996324 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6128a092-1455-4238-b4d6-57966665f1e3" containerName="payload-pre-processing" Oct 30 01:17:25.004883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:24.996333 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe31be1-f392-4843-9ec7-6602fb106b53" containerName="payload-processing" Oct 30 01:17:25.034729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.034695 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:17:25.035101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.035081 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.079173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.079144 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6d5b8-dockercfg-dvjfs\"" Oct 30 01:17:25.080209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.079856 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6d5b8\"" Oct 30 01:17:25.130149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.130116 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:17:25.130503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.130479 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.130594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.130582 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456rx\" (UniqueName: \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.134629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.134610 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.156313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.155921 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:17:25.231642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.231605 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456rx\" (UniqueName: \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.232009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.231984 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.232139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.232080 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2nc8\" (UniqueName: \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.232139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.232122 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.232702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.232680 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.254253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.254228 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456rx\" (UniqueName: \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\") pod \"payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.333449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.333378 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2nc8\" (UniqueName: \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.333449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.333416 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.334140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.334121 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.355478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.355449 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2nc8\" (UniqueName: \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\") pod \"payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.397002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.396984 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:25.448047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.448004 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:25.582169 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.582136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.701975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.701948 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:17:25.703359 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:17:25.703328 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c5c82c_2284_428c_beec_e54ea99ab295.slice/crio-78cf497285cd7560f199102509ec4f6222878cb499dce2d502876049ea4a46d2 WatchSource:0}: Error finding container 78cf497285cd7560f199102509ec4f6222878cb499dce2d502876049ea4a46d2: Status 404 returned error can't find the container with id 78cf497285cd7560f199102509ec4f6222878cb499dce2d502876049ea4a46d2 Oct 30 01:17:25.823636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:25.823556 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:17:25.827793 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:17:25.827669 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5b3681_820c_4666_a85f_24bbfd170aa1.slice/crio-80f1b93780bbcc1600cfb3060f8f89aad5d1c96e3f3a54c61b324e38ed335a2d WatchSource:0}: Error finding container 80f1b93780bbcc1600cfb3060f8f89aad5d1c96e3f3a54c61b324e38ed335a2d: Status 404 returned error can't find the container with id 80f1b93780bbcc1600cfb3060f8f89aad5d1c96e3f3a54c61b324e38ed335a2d Oct 30 01:17:26.097526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.097423 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" event={"ID":"45c5c82c-2284-428c-beec-e54ea99ab295","Type":"ContainerStarted","Data":"08e568f46125f6bf04dd4633225ded783b0f749a53d0f24a14ee7dc0f9296de0"} Oct 30 01:17:26.097526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.097490 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" event={"ID":"45c5c82c-2284-428c-beec-e54ea99ab295","Type":"ContainerStarted","Data":"78cf497285cd7560f199102509ec4f6222878cb499dce2d502876049ea4a46d2"} Oct 30 01:17:26.098401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.098273 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:26.100667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.100619 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" event={"ID":"9e5b3681-820c-4666-a85f-24bbfd170aa1","Type":"ContainerStarted","Data":"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05"} Oct 30 01:17:26.100667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.100662 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" event={"ID":"9e5b3681-820c-4666-a85f-24bbfd170aa1","Type":"ContainerStarted","Data":"80f1b93780bbcc1600cfb3060f8f89aad5d1c96e3f3a54c61b324e38ed335a2d"} Oct 30 01:17:26.100927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.100684 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:26.143421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.143372 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" podStartSLOduration=2.143355332 podStartE2EDuration="2.143355332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:26.142094061 +0000 UTC m=+2698.382329173" watchObservedRunningTime="2026-10-30 01:17:26.143355332 +0000 UTC m=+2698.383590447" Oct 30 01:17:26.185094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.185041 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" podStartSLOduration=1.185024099 podStartE2EDuration="1.185024099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:26.181371863 +0000 UTC m=+2698.421606974" watchObservedRunningTime="2026-10-30 01:17:26.185024099 +0000 UTC m=+2698.425259219" Oct 30 01:17:26.237826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.237780 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.380316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.380288 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.518393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.518324 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.875048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:26.875022 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:28.221820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:28.221790 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:28.663659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:28.663416 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:17:28.735618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:28.735431 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:28.852195 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:28.852145 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:17:29.354136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:29.354097 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.834322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:29.834248 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.984071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:29.984039 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.162967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.161924 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.324346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.324313 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.497041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.496955 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.615973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.615949 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:17:30.643420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.643401 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:17:30.797617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.797541 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 01:17:30.797846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.797819 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" containerID="cri-o://ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5" gracePeriod=30 Oct 30 01:17:30.865429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:30.865399 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:31.354979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:31.354944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:31.761918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:31.761801 5938 scope.go:122] "RemoveContainer" containerID="fdea9eb7619f679b61a9347e1f4a8e6f795f62766bc31b1703e28304bedbca13" Oct 30 01:17:32.436917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:32.436879 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.281463 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:33.281433 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.424086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:33.424052 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.629048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:33.629017 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.817883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:33.817360 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.956886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:33.956797 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:35.005879 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.005833 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:35.502181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.502144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:35.584515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.584487 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:17:35.590716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.590696 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.608258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.608237 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:17:35.615583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.615557 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b73232\"" Oct 30 01:17:35.654563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.654532 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:35.667717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.667686 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.667848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.667730 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rnz\" (UniqueName: \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.778737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.778660 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.793825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.793795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.794223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.794200 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rnz\" (UniqueName: \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.843429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.843395 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rnz\" (UniqueName: \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\") pod \"payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:35.907357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:35.907330 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:36.130833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.130808 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:17:36.207613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207587 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207709 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207816 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207881 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.207995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207876 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:36.207995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.208224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.207975 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:36.208224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208056 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.208224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208131 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\") pod \"3bd77300-19e2-4629-8e82-5498fbcabb5b\" (UID: \"3bd77300-19e2-4629-8e82-5498fbcabb5b\") " Oct 30 01:17:36.208224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208202 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:36.208529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208499 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:36.208529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208513 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:36.208789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208766 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.208898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208794 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.208898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208808 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.208898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208822 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.208898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.208835 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd77300-19e2-4629-8e82-5498fbcabb5b-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.210267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.210238 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:36.210485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.210457 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:36.210579 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.210526 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "kube-api-access-psf8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:36.210579 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.210562 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo" pod "3bd77300-19e2-4629-8e82-5498fbcabb5b" (UID: "3bd77300-19e2-4629-8e82-5498fbcabb5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:36.215696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.215678 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:17:36.219362 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:17:36.219341 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b15fc60_daef_4fe3_8f63_016a7011a891.slice/crio-6dcd9be524c6ffa905e97822bd52307e78cf98eb0a855d1cd94273439e300813 WatchSource:0}: Error finding container 6dcd9be524c6ffa905e97822bd52307e78cf98eb0a855d1cd94273439e300813: Status 404 returned error can't find the container with id 6dcd9be524c6ffa905e97822bd52307e78cf98eb0a855d1cd94273439e300813 Oct 30 01:17:36.260114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.260089 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 01:17:36.309929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.309907 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psf8h\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-kube-api-access-psf8h\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.309929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.309928 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.310126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.309938 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.310126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.309946 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd77300-19e2-4629-8e82-5498fbcabb5b-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:36.367941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.367237 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" event={"ID":"9b15fc60-daef-4fe3-8f63-016a7011a891","Type":"ContainerStarted","Data":"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a"} Oct 30 01:17:36.367941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.367317 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" event={"ID":"9b15fc60-daef-4fe3-8f63-016a7011a891","Type":"ContainerStarted","Data":"6dcd9be524c6ffa905e97822bd52307e78cf98eb0a855d1cd94273439e300813"} Oct 30 01:17:36.367941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.367374 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:36.369500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.369475 5938 generic.go:356] "Generic (PLEG): container finished" podID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerID="ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5" exitCode=0 Oct 30 01:17:36.369604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.369536 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" Oct 30 01:17:36.369604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.369561 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerDied","Data":"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5"} Oct 30 01:17:36.369712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.369612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9" event={"ID":"3bd77300-19e2-4629-8e82-5498fbcabb5b","Type":"ContainerDied","Data":"b24c8490fcc4dcfab0df97f3a294bbd59e7e9d8293d69711706d428e7ddaafca"} Oct 30 01:17:36.369712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.369642 5938 scope.go:122] "RemoveContainer" containerID="ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5" Oct 30 01:17:36.381130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.381114 5938 scope.go:122] "RemoveContainer" containerID="a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6" Oct 30 01:17:36.393648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.393629 5938 scope.go:122] "RemoveContainer" containerID="ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5" Oct 30 01:17:36.394523 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:36.394496 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5\": container with ID starting with ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5 not found: ID does not exist" containerID="ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5" Oct 30 01:17:36.394693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.394533 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5"} err="failed to get container status \"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5\": rpc error: code = NotFound desc = could not find container \"ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5\": container with ID starting with ca64923fd73317e3d0ce791b38041d8f7cd778595b440a6bec7c5d457d379ed5 not found: ID does not exist" Oct 30 01:17:36.394693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.394557 5938 scope.go:122] "RemoveContainer" containerID="a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6" Oct 30 01:17:36.395446 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:36.395419 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6\": container with ID starting with a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6 not found: ID does not exist" containerID="a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6" Oct 30 01:17:36.395677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.395646 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6"} err="failed to get container status \"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6\": rpc error: code = NotFound desc = could not find container \"a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6\": container with ID starting with a5c1438bb5f8a6f7204b5803a02d1edea936aae8f11195ccc2689aa8d9ccf9d6 not found: ID does not exist" Oct 30 01:17:36.411371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.411329 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" podStartSLOduration=1.411317585 podStartE2EDuration="1.411317585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:36.406619984 +0000 UTC m=+2708.646855098" watchObservedRunningTime="2026-10-30 01:17:36.411317585 +0000 UTC m=+2708.651552698" Oct 30 01:17:36.415164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.415136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.443439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.443415 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 01:17:36.452946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.452921 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-ngpk9"] Oct 30 01:17:36.983484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:36.983457 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.114229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:37.114206 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:17:37.878708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:37.878667 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" path="/var/lib/kubelet/pods/3bd77300-19e2-4629-8e82-5498fbcabb5b/volumes" Oct 30 01:17:38.123287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:38.123247 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:17:38.710262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:38.710214 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:40.840686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:40.840655 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.037277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:41.037240 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.219297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:41.219245 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.835856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:41.835804 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.043934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:42.043848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.211996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:42.211964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.353792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:42.353761 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.563627 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:42.563537 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.797297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:42.795930 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:43.691987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:43.691946 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" podUID="56edaa83-d578-4e8b-a09d-89d53835e246" containerName="main" containerID="cri-o://66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c" gracePeriod=45 Oct 30 01:17:43.974767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:43.974741 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:17:44.084695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.084663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\" (UniqueName: \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.084834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.084715 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.084834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.084766 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\" (UniqueName: \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.084985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.084949 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.085052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085002 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.085109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085053 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\") pod \"56edaa83-d578-4e8b-a09d-89d53835e246\" (UID: \"56edaa83-d578-4e8b-a09d-89d53835e246\") " Oct 30 01:17:44.085429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085406 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.085585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085423 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.085585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085564 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.085859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085817 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.085859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085844 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.085859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.085857 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.087383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.087357 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:44.087509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.087487 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:44.087624 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.087549 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz" pod "56edaa83-d578-4e8b-a09d-89d53835e246" (UID: "56edaa83-d578-4e8b-a09d-89d53835e246"). InnerVolumeSpecName "kube-api-access-6vddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:44.187235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.187205 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vddz\" (UniqueName: \"kubernetes.io/projected/56edaa83-d578-4e8b-a09d-89d53835e246-kube-api-access-6vddz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.187235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.187228 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56edaa83-d578-4e8b-a09d-89d53835e246-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.187235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.187238 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56edaa83-d578-4e8b-a09d-89d53835e246-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:44.436965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.436941 5938 generic.go:356] "Generic (PLEG): container finished" podID="56edaa83-d578-4e8b-a09d-89d53835e246" containerID="66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c" exitCode=0 Oct 30 01:17:44.437105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.436991 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" Oct 30 01:17:44.437105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.437019 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" event={"ID":"56edaa83-d578-4e8b-a09d-89d53835e246","Type":"ContainerDied","Data":"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c"} Oct 30 01:17:44.437105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.437056 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g" event={"ID":"56edaa83-d578-4e8b-a09d-89d53835e246","Type":"ContainerDied","Data":"6b1f4d633d11817259f942a1abd42813c928bf5282b92e1a75fcb10b8ee26fdd"} Oct 30 01:17:44.437105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.437075 5938 scope.go:122] "RemoveContainer" containerID="66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c" Oct 30 01:17:44.448536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.448519 5938 scope.go:122] "RemoveContainer" containerID="66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c" Oct 30 01:17:44.448815 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:44.448792 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c\": container with ID starting with 66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c not found: ID does not exist" containerID="66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c" Oct 30 01:17:44.448887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.448826 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c"} err="failed to get container status \"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c\": rpc error: code = NotFound desc = could not find container \"66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c\": container with ID starting with 66d3b980667d906e041c606d6baaedc6f86aae8771f8d7272cafb8ddd4d5c40c not found: ID does not exist" Oct 30 01:17:44.489015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.488945 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:17:44.500367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.500333 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/rate-test-model-7f92f6-kserve-655b9f7-s9k6g"] Oct 30 01:17:44.592019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:44.591994 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:45.878659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:45.878627 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56edaa83-d578-4e8b-a09d-89d53835e246" path="/var/lib/kubelet/pods/56edaa83-d578-4e8b-a09d-89d53835e246/volumes" Oct 30 01:17:45.997676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:45.997647 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:46.144977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:46.144905 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:46.632712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:46.632672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.381438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.381407 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:17:47.678914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.678823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.759112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759082 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:17:47.759896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759853 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.759896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759883 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.760023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759917 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.760023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759924 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.760023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759939 5938 cpu_manager.go:414] "containerMap: removing container" podUID="56edaa83-d578-4e8b-a09d-89d53835e246" containerName="main" Oct 30 01:17:47.760023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.759944 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56edaa83-d578-4e8b-a09d-89d53835e246" containerName="main" Oct 30 01:17:47.760150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.760068 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.760150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.760081 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd77300-19e2-4629-8e82-5498fbcabb5b" containerName="istio-proxy" Oct 30 01:17:47.760150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.760089 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="56edaa83-d578-4e8b-a09d-89d53835e246" containerName="main" Oct 30 01:17:47.775339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.775317 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:47.781530 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.781505 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:17:47.794229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.794207 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-16c30f\"" Oct 30 01:17:47.813333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.813307 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:17:47.818723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.818703 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:47.828034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.828004 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.828408 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.828380 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:47.828694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.828671 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77qc\" (UniqueName: \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:47.833202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.833182 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-16c30f-dockercfg-vp7bw\"" Oct 30 01:17:47.836581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.836558 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:17:47.957108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.957005 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77qc\" (UniqueName: \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:47.957411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.957384 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:47.957701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.957670 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:47.958136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.958100 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnfn\" (UniqueName: \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:47.959711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:47.959687 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:48.001545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.001516 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77qc\" (UniqueName: \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\") pod \"payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:48.060048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.060010 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnfn\" (UniqueName: \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:48.060235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.060176 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:48.060874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.060846 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:48.082410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.082386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnfn\" (UniqueName: \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:48.095068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.095043 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:48.139015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.138990 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:48.408274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.408251 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:17:48.409560 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:17:48.409524 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c253e9c_ee91_4361_85bf_42be96b6ed73.slice/crio-094fba61c76ae4a2b3273a8cb84f4d60c6b86216688930d6c0231062a05369e1 WatchSource:0}: Error finding container 094fba61c76ae4a2b3273a8cb84f4d60c6b86216688930d6c0231062a05369e1: Status 404 returned error can't find the container with id 094fba61c76ae4a2b3273a8cb84f4d60c6b86216688930d6c0231062a05369e1 Oct 30 01:17:48.476494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.476464 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" event={"ID":"3c253e9c-ee91-4361-85bf-42be96b6ed73","Type":"ContainerStarted","Data":"094fba61c76ae4a2b3273a8cb84f4d60c6b86216688930d6c0231062a05369e1"} Oct 30 01:17:48.477652 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.477628 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:17:48.478593 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:17:48.478523 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3930f0d4_7c13_49f7_9deb_9fc654c9b46f.slice/crio-5de2b375e089522ee7a629c12fad08753b7e9a4c1dc7c5554ca41ec4543cc2b7 WatchSource:0}: Error finding container 5de2b375e089522ee7a629c12fad08753b7e9a4c1dc7c5554ca41ec4543cc2b7: Status 404 returned error can't find the container with id 5de2b375e089522ee7a629c12fad08753b7e9a4c1dc7c5554ca41ec4543cc2b7 Oct 30 01:17:48.644209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:48.644180 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:49.486728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.486688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" event={"ID":"3c253e9c-ee91-4361-85bf-42be96b6ed73","Type":"ContainerStarted","Data":"f6248097720a9e75ef3ae58b8b6bb12bf63cb7d09f0da87d823be26372aa72a0"} Oct 30 01:17:49.487225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.486932 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:17:49.488383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.488360 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" event={"ID":"3930f0d4-7c13-49f7-9deb-9fc654c9b46f","Type":"ContainerStarted","Data":"00cfac1c8bf2f13c7191aa67c60ef25d0b0f3c0edffa84a094f2b3765910b7d7"} Oct 30 01:17:49.488501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.488402 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" event={"ID":"3930f0d4-7c13-49f7-9deb-9fc654c9b46f","Type":"ContainerStarted","Data":"5de2b375e089522ee7a629c12fad08753b7e9a4c1dc7c5554ca41ec4543cc2b7"} Oct 30 01:17:49.488501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.488456 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:17:49.528314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.528271 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" podStartSLOduration=2.528257117 podStartE2EDuration="2.528257117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:49.524804553 +0000 UTC m=+2721.765061927" watchObservedRunningTime="2026-10-30 01:17:49.528257117 +0000 UTC m=+2721.768492230" Oct 30 01:17:49.563830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:49.563787 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" podStartSLOduration=2.563774882 podStartE2EDuration="2.563774882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:49.562599598 +0000 UTC m=+2721.802834736" watchObservedRunningTime="2026-10-30 01:17:49.563774882 +0000 UTC m=+2721.804009995" Oct 30 01:17:50.389039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:50.389002 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.284879 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.284804 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" containerID="cri-o://0b48937cd557caa042d1ceb1f004f4b3ccc3b81642346a700bac99fcc0583b49" gracePeriod=45 Oct 30 01:17:51.383975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.383938 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": dial tcp 10.134.0.124:8000: connect: connection refused" Oct 30 01:17:51.506364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.506328 5938 generic.go:356] "Generic (PLEG): container finished" podID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerID="0b48937cd557caa042d1ceb1f004f4b3ccc3b81642346a700bac99fcc0583b49" exitCode=0 Oct 30 01:17:51.506539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.506405 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" event={"ID":"c3487d6f-772d-4943-8244-0b2690c4b0f5","Type":"ContainerDied","Data":"0b48937cd557caa042d1ceb1f004f4b3ccc3b81642346a700bac99fcc0583b49"} Oct 30 01:17:51.556299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.556239 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 01:17:51.607282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607251 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.607459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607359 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.607459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607419 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.607586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607496 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.607586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607545 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\" (UniqueName: \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.607697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607637 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:51.608132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607769 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\") pod \"c3487d6f-772d-4943-8244-0b2690c4b0f5\" (UID: \"c3487d6f-772d-4943-8244-0b2690c4b0f5\") " Oct 30 01:17:51.608132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.607914 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:51.608454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.608305 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:51.610472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.608658 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:51.610472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.608682 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:51.610472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.608695 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:51.611068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.611044 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:51.611239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.611218 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "kube-api-access-959kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:51.612364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.612338 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs" pod "c3487d6f-772d-4943-8244-0b2690c4b0f5" (UID: "c3487d6f-772d-4943-8244-0b2690c4b0f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:51.710477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.710447 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3487d6f-772d-4943-8244-0b2690c4b0f5-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:51.710477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.710477 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-959kp\" (UniqueName: \"kubernetes.io/projected/c3487d6f-772d-4943-8244-0b2690c4b0f5-kube-api-access-959kp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:51.710687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:51.710486 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3487d6f-772d-4943-8244-0b2690c4b0f5-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:17:52.109651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.109606 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:52.515741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.515714 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" Oct 30 01:17:52.516248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.515760 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt" event={"ID":"c3487d6f-772d-4943-8244-0b2690c4b0f5","Type":"ContainerDied","Data":"ef354f844382adbe18f8c2a5a10b27d027a0b03cdae13f2c01bdcb8dddef3aa0"} Oct 30 01:17:52.516248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.515809 5938 scope.go:122] "RemoveContainer" containerID="0b48937cd557caa042d1ceb1f004f4b3ccc3b81642346a700bac99fcc0583b49" Oct 30 01:17:52.565591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.565566 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 01:17:52.577133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:52.577109 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-2-w1-29be73-kserve-869dbf6b89-r49mt"] Oct 30 01:17:53.877656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:53.877626 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" path="/var/lib/kubelet/pods/c3487d6f-772d-4943-8244-0b2690c4b0f5/volumes" Oct 30 01:17:54.022784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:54.022754 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:55.756656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:55.756618 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:57.485197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:57.485163 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:58.834818 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:17:58.834782 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:17:59.377967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:17:59.377929 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:00.500438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:00.500403 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:18:00.500919 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:00.500485 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:18:01.355404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:01.355371 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:01.727890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:01.727794 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:18:01.759479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:01.759442 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:18:01.898743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:01.898710 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.046055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:02.045976 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.189350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:02.189321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:03.607941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:03.607910 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:05.476107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:05.476072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:06.044497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:06.044466 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:06.729010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:06.728971 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" podUID="dc33f001-ee60-48d6-b274-7f4569d66dd4" containerName="payload-processing" containerID="cri-o://cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7" gracePeriod=25 Oct 30 01:18:06.760521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:06.760478 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" podUID="a7831abe-18da-43bc-81d7-700c42b2a306" containerName="payload-pre-processing" containerID="cri-o://1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c" gracePeriod=25 Oct 30 01:18:07.038601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.038575 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:18:07.078344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.078322 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:18:07.206384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206350 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\") pod \"a7831abe-18da-43bc-81d7-700c42b2a306\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " Oct 30 01:18:07.206587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206432 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\") pod \"dc33f001-ee60-48d6-b274-7f4569d66dd4\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " Oct 30 01:18:07.206669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206618 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\" (UniqueName: \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\") pod \"a7831abe-18da-43bc-81d7-700c42b2a306\" (UID: \"a7831abe-18da-43bc-81d7-700c42b2a306\") " Oct 30 01:18:07.206669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206662 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\" (UniqueName: \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\") pod \"dc33f001-ee60-48d6-b274-7f4569d66dd4\" (UID: \"dc33f001-ee60-48d6-b274-7f4569d66dd4\") " Oct 30 01:18:07.206803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206757 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume" pod "a7831abe-18da-43bc-81d7-700c42b2a306" (UID: "a7831abe-18da-43bc-81d7-700c42b2a306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:07.206946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.206859 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume" pod "dc33f001-ee60-48d6-b274-7f4569d66dd4" (UID: "dc33f001-ee60-48d6-b274-7f4569d66dd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:07.207225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.207207 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7831abe-18da-43bc-81d7-700c42b2a306-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:07.207298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.207230 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc33f001-ee60-48d6-b274-7f4569d66dd4-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:07.208787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.208768 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5" pod "a7831abe-18da-43bc-81d7-700c42b2a306" (UID: "a7831abe-18da-43bc-81d7-700c42b2a306"). InnerVolumeSpecName "kube-api-access-z42g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:07.208884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.208797 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v" pod "dc33f001-ee60-48d6-b274-7f4569d66dd4" (UID: "dc33f001-ee60-48d6-b274-7f4569d66dd4"). InnerVolumeSpecName "kube-api-access-9rk5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:07.265919 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.265841 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:07.309605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.309572 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42g5\" (UniqueName: \"kubernetes.io/projected/a7831abe-18da-43bc-81d7-700c42b2a306-kube-api-access-z42g5\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:07.309846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.309823 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rk5v\" (UniqueName: \"kubernetes.io/projected/dc33f001-ee60-48d6-b274-7f4569d66dd4-kube-api-access-9rk5v\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:07.632573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.632537 5938 generic.go:356] "Generic (PLEG): container finished" podID="dc33f001-ee60-48d6-b274-7f4569d66dd4" containerID="cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7" exitCode=0 Oct 30 01:18:07.632761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.632595 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" Oct 30 01:18:07.632761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.632624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" event={"ID":"dc33f001-ee60-48d6-b274-7f4569d66dd4","Type":"ContainerDied","Data":"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7"} Oct 30 01:18:07.632761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.632714 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh" event={"ID":"dc33f001-ee60-48d6-b274-7f4569d66dd4","Type":"ContainerDied","Data":"e8f19a66d67146d150df221bee291c1ab72101bf9157faec1659bd76bf54a13f"} Oct 30 01:18:07.632761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.632742 5938 scope.go:122] "RemoveContainer" containerID="cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7" Oct 30 01:18:07.634313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.634287 5938 generic.go:356] "Generic (PLEG): container finished" podID="a7831abe-18da-43bc-81d7-700c42b2a306" containerID="1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c" exitCode=0 Oct 30 01:18:07.634451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.634360 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" Oct 30 01:18:07.634451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.634389 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" event={"ID":"a7831abe-18da-43bc-81d7-700c42b2a306","Type":"ContainerDied","Data":"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c"} Oct 30 01:18:07.634550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.634451 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7" event={"ID":"a7831abe-18da-43bc-81d7-700c42b2a306","Type":"ContainerDied","Data":"787eeda20e80e00538e078b01dde2e9f59e27a2e09cc733e7370e1591024f321"} Oct 30 01:18:07.646759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.646673 5938 scope.go:122] "RemoveContainer" containerID="cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7" Oct 30 01:18:07.647062 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:18:07.647039 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7\": container with ID starting with cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7 not found: ID does not exist" containerID="cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7" Oct 30 01:18:07.647162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.647070 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7"} err="failed to get container status \"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7\": rpc error: code = NotFound desc = could not find container \"cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7\": container with ID starting with cb86f6b5e6c1b16115aa02dcb62e85a77a53537a985b7e0d2a36f584b8f63dc7 not found: ID does not exist" Oct 30 01:18:07.647162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.647090 5938 scope.go:122] "RemoveContainer" containerID="1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c" Oct 30 01:18:07.656677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.656661 5938 scope.go:122] "RemoveContainer" containerID="1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c" Oct 30 01:18:07.656932 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:18:07.656910 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c\": container with ID starting with 1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c not found: ID does not exist" containerID="1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c" Oct 30 01:18:07.657010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.656941 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c"} err="failed to get container status \"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c\": rpc error: code = NotFound desc = could not find container \"1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c\": container with ID starting with 1ea47ffc124dc6b2d89055d2d7ea9ce94edb6fc3ab885d9e4b24437d9d75fa2c not found: ID does not exist" Oct 30 01:18:07.683190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.683170 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:18:07.687210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.687191 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-68775d7859-h8jm7"] Oct 30 01:18:07.718705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.718683 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:18:07.722980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.722959 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-75767f6bf6-ktwwh"] Oct 30 01:18:07.878315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.878288 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7831abe-18da-43bc-81d7-700c42b2a306" path="/var/lib/kubelet/pods/a7831abe-18da-43bc-81d7-700c42b2a306/volumes" Oct 30 01:18:07.878713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:07.878698 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc33f001-ee60-48d6-b274-7f4569d66dd4" path="/var/lib/kubelet/pods/dc33f001-ee60-48d6-b274-7f4569d66dd4/volumes" Oct 30 01:18:09.430635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:09.430586 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:11.276835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:11.276798 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:13.045448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:13.045386 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:13.208855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:13.208823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:13.357912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:13.357880 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:14.380574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:14.380544 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:14.846970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:14.846852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:17.099587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:17.099555 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:17.261785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:17.261737 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:17.397189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:17.397159 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.966346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:18.966308 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:20.352634 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:20.352596 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:18:20.488908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:20.488858 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:20.632340 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:20.632249 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:20.783631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:20.783603 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:20.934545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:20.934473 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.097856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:21.097820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:22.547732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:22.547669 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:24.870930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:24.870899 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.353442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.353411 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" podUID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" containerName="payload-pre-processing" containerID="cri-o://47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206" gracePeriod=25 Oct 30 01:18:25.673965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.673938 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:18:25.777752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.777718 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\") pod \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " Oct 30 01:18:25.777984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.777785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\" (UniqueName: \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\") pod \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\" (UID: \"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8\") " Oct 30 01:18:25.778228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.778199 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume" pod "34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" (UID: "34ae5323-5bfc-49ae-bf2f-9b9edea05fc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:25.778697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.778672 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:25.780158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.780132 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl" pod "34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" (UID: "34ae5323-5bfc-49ae-bf2f-9b9edea05fc8"). InnerVolumeSpecName "kube-api-access-zhhcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:25.818990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.818960 5938 generic.go:356] "Generic (PLEG): container finished" podID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" containerID="47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206" exitCode=0 Oct 30 01:18:25.819117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.819010 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" Oct 30 01:18:25.819117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.819039 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" event={"ID":"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8","Type":"ContainerDied","Data":"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206"} Oct 30 01:18:25.819117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.819092 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5" event={"ID":"34ae5323-5bfc-49ae-bf2f-9b9edea05fc8","Type":"ContainerDied","Data":"9ec3bc01c8f9c2b4e6b1d46a59361bfa403ac2758f88625e0a1444d2bc8f31f9"} Oct 30 01:18:25.819117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.819114 5938 scope.go:122] "RemoveContainer" containerID="47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206" Oct 30 01:18:25.829433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.829413 5938 scope.go:122] "RemoveContainer" containerID="47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206" Oct 30 01:18:25.829684 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:18:25.829663 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206\": container with ID starting with 47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206 not found: ID does not exist" containerID="47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206" Oct 30 01:18:25.829751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.829695 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206"} err="failed to get container status \"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206\": rpc error: code = NotFound desc = could not find container \"47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206\": container with ID starting with 47752bdfdf2f4d064973d618ebde8185c5c90037b586a4a510309c14d4167206 not found: ID does not exist" Oct 30 01:18:25.877750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.877725 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:18:25.878178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.877761 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-6f9b9d5fc5-dtcs5"] Oct 30 01:18:25.880697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:25.880668 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhcl\" (UniqueName: \"kubernetes.io/projected/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8-kube-api-access-zhhcl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:26.610367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:26.610334 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:27.303570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:27.303534 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv_bc520c15-f605-4f32-90dd-5e6d2a5d1e41/payload-processing/0.log" Oct 30 01:18:27.880682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:27.880648 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" path="/var/lib/kubelet/pods/34ae5323-5bfc-49ae-bf2f-9b9edea05fc8/volumes" Oct 30 01:18:27.980200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:27.980178 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz_0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b/payload-processing/0.log" Oct 30 01:18:28.169888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:28.169802 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz_0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b/payload-processing/0.log" Oct 30 01:18:28.260679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:28.260636 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:28.839268 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:18:28.839224 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:18:29.807484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.807440 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:18:29.808148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808129 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dc33f001-ee60-48d6-b274-7f4569d66dd4" containerName="payload-processing" Oct 30 01:18:29.808148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808145 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc33f001-ee60-48d6-b274-7f4569d66dd4" containerName="payload-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808183 5938 cpu_manager.go:414] "containerMap: removing container" podUID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" containerName="payload-pre-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808188 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" containerName="payload-pre-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808212 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808217 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808229 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a7831abe-18da-43bc-81d7-700c42b2a306" containerName="payload-pre-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808234 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7831abe-18da-43bc-81d7-700c42b2a306" containerName="payload-pre-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808343 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc33f001-ee60-48d6-b274-7f4569d66dd4" containerName="payload-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808355 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ae5323-5bfc-49ae-bf2f-9b9edea05fc8" containerName="payload-pre-processing" Oct 30 01:18:29.808359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808363 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3487d6f-772d-4943-8244-0b2690c4b0f5" containerName="main" Oct 30 01:18:29.808926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.808372 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7831abe-18da-43bc-81d7-700c42b2a306" containerName="payload-pre-processing" Oct 30 01:18:29.826961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.826942 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:29.831005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.830945 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:18:29.846315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.846291 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-16c30f-serving-cert\"" Oct 30 01:18:29.933316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.933283 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.950875 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.950840 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:29.951009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.950949 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfhr\" (UniqueName: \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:29.951070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.951048 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:29.951207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:29.951191 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.052152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.052121 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.052380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.052293 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.052466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.052418 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.052528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.052492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfhr\" (UniqueName: \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.053092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.053060 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.054955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.054932 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.055287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.055260 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.074797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.074748 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfhr\" (UniqueName: \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\") pod \"maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.162474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.162442 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:30.482389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.482361 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:18:30.483344 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:18:30.483318 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ccba01_1713_4d66_804b_25500dc2f52a.slice/crio-900bf805b2601467c423fb7843a56a6a21b15d14f5b829e773b33cfbb965b4fa WatchSource:0}: Error finding container 900bf805b2601467c423fb7843a56a6a21b15d14f5b829e773b33cfbb965b4fa: Status 404 returned error can't find the container with id 900bf805b2601467c423fb7843a56a6a21b15d14f5b829e773b33cfbb965b4fa Oct 30 01:18:30.485303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.485282 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:18:30.870423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:30.870395 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" event={"ID":"73ccba01-1713-4d66-804b-25500dc2f52a","Type":"ContainerStarted","Data":"900bf805b2601467c423fb7843a56a6a21b15d14f5b829e773b33cfbb965b4fa"} Oct 30 01:18:31.880855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.880823 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" event={"ID":"73ccba01-1713-4d66-804b-25500dc2f52a","Type":"ContainerStarted","Data":"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa"} Oct 30 01:18:31.881285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.880916 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:31.882723 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.882698 5938 generic.go:356] "Generic (PLEG): container finished" podID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerID="731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228" exitCode=0 Oct 30 01:18:31.882836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.882759 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerDied","Data":"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228"} Oct 30 01:18:31.883098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.883083 5938 scope.go:122] "RemoveContainer" containerID="731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228" Oct 30 01:18:31.885267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.885241 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:31.885355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.885324 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:31.885419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.885375 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:31.924940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.924897 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" podStartSLOduration=2.557293866 podStartE2EDuration="2.924880624s" totalImagesPullingTime="367.586758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:31.923355377 +0000 UTC m=+2764.163590492" watchObservedRunningTime="2026-10-30 01:18:31.924880624 +0000 UTC m=+2764.165115736" Oct 30 01:18:31.951098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:31.951072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:32.582589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:32.582559 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:18:32.916221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:32.916184 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerStarted","Data":"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c"} Oct 30 01:18:33.660252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:33.660223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:33.667159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:33.667122 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:18:33.668497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:33.668468 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:18:33.668647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:33.668535 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Oct 30 01:18:34.667123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:34.667080 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:18:34.667506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:34.667138 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Oct 30 01:18:35.375476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:35.375446 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:35.667663 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:35.667580 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:18:35.667663 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:35.667635 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Oct 30 01:18:36.666899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:36.666840 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:18:36.667060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:36.666921 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Oct 30 01:18:37.078626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:37.078543 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:37.667811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:37.667772 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Oct 30 01:18:37.668003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:37.667829 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Oct 30 01:18:37.922095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:37.922017 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:18:38.509673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:38.509640 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:38.701081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:38.699486 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:18:38.701081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:38.700914 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:18:38.702629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:38.702605 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:18:39.222052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:39.222009 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:39.381500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:39.381463 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:39.818968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:39.818932 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:39.958423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:39.958396 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:40.524294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.524262 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:40.527999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.527976 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.536227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.536200 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:40.540303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.540281 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.542503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.542451 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29be73-openshift-default-dockercfg-xqn9v\"" Oct 30 01:18:40.550349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.550322 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:40.582692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.582659 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:40.625285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625250 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625295 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctb6\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625374 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625493 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625596 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.625742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625642 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.625742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.625843 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625753 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.625843 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625794 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.625843 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625820 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.626033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.626033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.625970 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.626149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.626081 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rj8d\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.626149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.626138 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.626262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.626169 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.626262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.626197 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.626378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.626270 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727262 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727290 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727313 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727351 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.727420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727376 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727414 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.727720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727531 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.727720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727564 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.727720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727589 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.727720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727636 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.727720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727670 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727717 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727756 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rj8d\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727796 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727809 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727817 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727839 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727927 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727941 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.727998 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728027 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728097 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728137 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctb6\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728683 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.728775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728693 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728780 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.728900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728780 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.729042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.728962 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.730221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.730201 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.730359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.730340 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.730519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.730503 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.731017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.730999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.750281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.750244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.750407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.750369 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.750645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.750625 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rj8d\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.750734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.750715 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctb6\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.844363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.844308 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:40.856539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.856518 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:40.984182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:40.984149 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:41.179336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.179305 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:41.185417 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:18:41.185384 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b17541_77ee_4fcf_ae99_c44536e72651.slice/crio-8c78a4ff0e41a6a44c0de0169aec3911a55abe55854aef3e73da5958eb03d1dd WatchSource:0}: Error finding container 8c78a4ff0e41a6a44c0de0169aec3911a55abe55854aef3e73da5958eb03d1dd: Status 404 returned error can't find the container with id 8c78a4ff0e41a6a44c0de0169aec3911a55abe55854aef3e73da5958eb03d1dd Oct 30 01:18:41.189385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.188450 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.189385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.188535 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.189385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.188580 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.189682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.189447 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:41.193322 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:18:41.193303 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a3ff7b_b71c_4c77_894b_d8176d91c4d2.slice/crio-d6d2269d069ec02ac29823b9ad7fb5fc7013faf7953f13d1baef6037e6c2661c WatchSource:0}: Error finding container d6d2269d069ec02ac29823b9ad7fb5fc7013faf7953f13d1baef6037e6c2661c: Status 404 returned error can't find the container with id d6d2269d069ec02ac29823b9ad7fb5fc7013faf7953f13d1baef6037e6c2661c Oct 30 01:18:41.202419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.202389 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.202527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.202467 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.202527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.202495 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:18:41.463712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:41.461957 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:42.039260 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.039225 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" event={"ID":"b0b17541-77ee-4fcf-ae99-c44536e72651","Type":"ContainerStarted","Data":"b75fd85016ff1501a0d6e1ae0fd92ad961f6590e50640dd898f4fe926e8f8b6b"} Oct 30 01:18:42.039817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.039280 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" event={"ID":"b0b17541-77ee-4fcf-ae99-c44536e72651","Type":"ContainerStarted","Data":"8c78a4ff0e41a6a44c0de0169aec3911a55abe55854aef3e73da5958eb03d1dd"} Oct 30 01:18:42.041140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.041112 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" event={"ID":"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2","Type":"ContainerStarted","Data":"69701c1003318881bd304b81678ca4d183753b51ce98b858a90f92631a3c4d06"} Oct 30 01:18:42.041261 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.041180 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" event={"ID":"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2","Type":"ContainerStarted","Data":"d6d2269d069ec02ac29823b9ad7fb5fc7013faf7953f13d1baef6037e6c2661c"} Oct 30 01:18:42.138692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.138633 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podStartSLOduration=2.1386153119999998 podStartE2EDuration="2.138615312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:42.118217973 +0000 UTC m=+2774.358453088" watchObservedRunningTime="2026-10-30 01:18:42.138615312 +0000 UTC m=+2774.378850425" Oct 30 01:18:42.745228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.745194 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:42.846887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.846103 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:42.846887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.846693 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Oct 30 01:18:42.846887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.846743 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Oct 30 01:18:42.873902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.870733 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:42.909886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.909337 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 01:18:42.909886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.909460 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 01:18:42.981341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:42.981298 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:43.845332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:43.845299 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Oct 30 01:18:43.845739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:43.845363 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Oct 30 01:18:43.857548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:43.857525 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 01:18:43.857733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:43.857714 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 01:18:43.910883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:43.907613 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:44.844842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:44.844798 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Oct 30 01:18:44.845075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:44.844897 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Oct 30 01:18:44.857199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:44.857175 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 01:18:44.857613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:44.857591 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 01:18:45.713334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.713297 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:45.848418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.848382 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Oct 30 01:18:45.848710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.848677 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Oct 30 01:18:45.860767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.860416 5938 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 01:18:45.860767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.860477 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 01:18:45.941907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:45.941852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.099966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.093229 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.241057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.240992 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podStartSLOduration=6.240969988 podStartE2EDuration="6.240969988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:42.167706061 +0000 UTC m=+2774.407941193" watchObservedRunningTime="2026-10-30 01:18:46.240969988 +0000 UTC m=+2778.481205102" Oct 30 01:18:46.260655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.260627 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:46.261142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.261111 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" containerID="cri-o://b75fd85016ff1501a0d6e1ae0fd92ad961f6590e50640dd898f4fe926e8f8b6b" gracePeriod=30 Oct 30 01:18:46.266049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.266017 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:46.266250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.266213 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" containerID="cri-o://69701c1003318881bd304b81678ca4d183753b51ce98b858a90f92631a3c4d06" gracePeriod=30 Oct 30 01:18:46.361595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.356305 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.873954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:46.873917 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:47.105886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:47.105342 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:47.683111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:47.683049 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:49.391392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:49.391351 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:51.233563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:51.233532 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:53.534298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:53.534252 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:54.133011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.132979 5938 generic.go:356] "Generic (PLEG): container finished" podID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerID="69701c1003318881bd304b81678ca4d183753b51ce98b858a90f92631a3c4d06" exitCode=0 Oct 30 01:18:54.133224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.133205 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" event={"ID":"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2","Type":"ContainerDied","Data":"69701c1003318881bd304b81678ca4d183753b51ce98b858a90f92631a3c4d06"} Oct 30 01:18:54.134905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.134849 5938 generic.go:356] "Generic (PLEG): container finished" podID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerID="b75fd85016ff1501a0d6e1ae0fd92ad961f6590e50640dd898f4fe926e8f8b6b" exitCode=0 Oct 30 01:18:54.135025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.134978 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" event={"ID":"b0b17541-77ee-4fcf-ae99-c44536e72651","Type":"ContainerDied","Data":"b75fd85016ff1501a0d6e1ae0fd92ad961f6590e50640dd898f4fe926e8f8b6b"} Oct 30 01:18:54.303439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.303417 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:54.308370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.308351 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:54.480474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480431 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.480669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480516 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.480669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480573 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.480669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.480873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480720 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.480873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480722 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.480873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480763 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.480873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480794 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480890 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.481116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480937 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.481116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.480990 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.481116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481054 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.481116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481108 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481115 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481153 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481229 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481245 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481307 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481337 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.481396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481365 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481414 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481458 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\") pod \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\" (UID: \"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2\") " Oct 30 01:18:54.481798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481511 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\") pod \"b0b17541-77ee-4fcf-ae99-c44536e72651\" (UID: \"b0b17541-77ee-4fcf-ae99-c44536e72651\") " Oct 30 01:18:54.481798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481517 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:54.481798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.481714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482265 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482296 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482310 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482323 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482337 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482349 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.482361 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b17541-77ee-4fcf-ae99-c44536e72651-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.483846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.483714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.483846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.483739 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.483846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.483833 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.484101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.483941 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.484626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.484597 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "kube-api-access-4rj8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:54.485198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485172 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:54.485198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485189 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:54.485361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485218 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:54.485361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485258 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "kube-api-access-bctb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:54.485361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo" pod "b0b17541-77ee-4fcf-ae99-c44536e72651" (UID: "b0b17541-77ee-4fcf-ae99-c44536e72651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:18:54.485794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.485777 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo" pod "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" (UID: "e9a3ff7b-b71c-4c77-894b-d8176d91c4d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:18:54.583258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583228 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583253 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583263 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rj8d\" (UniqueName: \"kubernetes.io/projected/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-kube-api-access-4rj8d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583274 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583283 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583290 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583299 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583308 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583315 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctb6\" (UniqueName: \"kubernetes.io/projected/b0b17541-77ee-4fcf-ae99-c44536e72651-kube-api-access-bctb6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583323 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:54.583752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:54.583330 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b17541-77ee-4fcf-ae99-c44536e72651-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:18:55.144329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.144299 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" Oct 30 01:18:55.144509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.144338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk" event={"ID":"b0b17541-77ee-4fcf-ae99-c44536e72651","Type":"ContainerDied","Data":"8c78a4ff0e41a6a44c0de0169aec3911a55abe55854aef3e73da5958eb03d1dd"} Oct 30 01:18:55.144509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.144394 5938 scope.go:122] "RemoveContainer" containerID="b75fd85016ff1501a0d6e1ae0fd92ad961f6590e50640dd898f4fe926e8f8b6b" Oct 30 01:18:55.145924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.145900 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" event={"ID":"e9a3ff7b-b71c-4c77-894b-d8176d91c4d2","Type":"ContainerDied","Data":"d6d2269d069ec02ac29823b9ad7fb5fc7013faf7953f13d1baef6037e6c2661c"} Oct 30 01:18:55.146051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.145904 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b" Oct 30 01:18:55.155838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.155726 5938 scope.go:122] "RemoveContainer" containerID="69701c1003318881bd304b81678ca4d183753b51ce98b858a90f92631a3c4d06" Oct 30 01:18:55.206656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.206634 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:55.220638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.219257 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-jwjhk"] Oct 30 01:18:55.238827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.238772 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:55.243397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.243170 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:55.248191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.248169 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-9c28b"] Oct 30 01:18:55.878697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.878666 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" path="/var/lib/kubelet/pods/b0b17541-77ee-4fcf-ae99-c44536e72651/volumes" Oct 30 01:18:55.879356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:55.879330 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" path="/var/lib/kubelet/pods/e9a3ff7b-b71c-4c77-894b-d8176d91c4d2/volumes" Oct 30 01:18:56.938732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:56.938697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:58.299916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.299882 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:58.467557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.467529 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:58.823497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.823463 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:18:58.824549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824522 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" Oct 30 01:18:58.824549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824548 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" Oct 30 01:18:58.824712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824584 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" Oct 30 01:18:58.824712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824593 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" Oct 30 01:18:58.824787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824773 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b17541-77ee-4fcf-ae99-c44536e72651" containerName="istio-proxy" Oct 30 01:18:58.824823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.824802 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a3ff7b-b71c-4c77-894b-d8176d91c4d2" containerName="istio-proxy" Oct 30 01:18:58.828355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.828329 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.837708 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:18:58.837681 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:18:58.844956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.844598 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh2g\" (UniqueName: \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.844956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.844677 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.851824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.851796 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:18:58.852315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.852293 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812ba2\"" Oct 30 01:18:58.852447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.852374 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812ba2-dockercfg-h2q6l\"" Oct 30 01:18:58.947327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.947292 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh2g\" (UniqueName: \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.947518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.947367 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.948146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.948125 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:58.970282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:58.970255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh2g\" (UniqueName: \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\") pod \"payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:59.143158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:59.143130 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:18:59.449227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:59.449204 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:18:59.450797 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:18:59.450768 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19481de2_9d24_474e_abd3_82c1bfa0b2f4.slice/crio-5cf3453ba2f1f6e9f73aab4bfd7587e2f26f6f9320689c8d856f9ad160dea83e WatchSource:0}: Error finding container 5cf3453ba2f1f6e9f73aab4bfd7587e2f26f6f9320689c8d856f9ad160dea83e: Status 404 returned error can't find the container with id 5cf3453ba2f1f6e9f73aab4bfd7587e2f26f6f9320689c8d856f9ad160dea83e Oct 30 01:18:59.483505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:18:59.483466 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.189738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:00.189704 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" event={"ID":"19481de2-9d24-474e-abd3-82c1bfa0b2f4","Type":"ContainerStarted","Data":"0795254e8effc1c21554934a9bd92f3b6586ba857af5b5f3d72783bf3fde0afe"} Oct 30 01:19:00.189928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:00.189764 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" event={"ID":"19481de2-9d24-474e-abd3-82c1bfa0b2f4","Type":"ContainerStarted","Data":"5cf3453ba2f1f6e9f73aab4bfd7587e2f26f6f9320689c8d856f9ad160dea83e"} Oct 30 01:19:00.189928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:00.189819 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:19:00.232772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:00.232719 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" podStartSLOduration=2.232701895 podStartE2EDuration="2.232701895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:00.228808821 +0000 UTC m=+2792.469043928" watchObservedRunningTime="2026-10-30 01:19:00.232701895 +0000 UTC m=+2792.472937009" Oct 30 01:19:02.066874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:02.066823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:03.700153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:03.700121 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.583375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:05.583342 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.926342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:05.926310 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:19:06.098359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:06.098318 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.248705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:06.248620 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.408545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:06.408516 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:07.366459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:07.366428 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:09.777913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:09.777787 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:10.927384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:10.927335 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" podUID="9b15fc60-daef-4fe3-8f63-016a7011a891" containerName="payload-processing" containerID="cri-o://b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a" gracePeriod=25 Oct 30 01:19:11.198736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.198711 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:19:11.215742 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.215715 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:19:11.286644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.286608 5938 generic.go:356] "Generic (PLEG): container finished" podID="9b15fc60-daef-4fe3-8f63-016a7011a891" containerID="b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a" exitCode=0 Oct 30 01:19:11.286829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.286690 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" event={"ID":"9b15fc60-daef-4fe3-8f63-016a7011a891","Type":"ContainerDied","Data":"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a"} Oct 30 01:19:11.286829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.286759 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" event={"ID":"9b15fc60-daef-4fe3-8f63-016a7011a891","Type":"ContainerDied","Data":"6dcd9be524c6ffa905e97822bd52307e78cf98eb0a855d1cd94273439e300813"} Oct 30 01:19:11.286829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.286792 5938 scope.go:122] "RemoveContainer" containerID="b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a" Oct 30 01:19:11.287349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.287324 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg" Oct 30 01:19:11.300060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.300035 5938 scope.go:122] "RemoveContainer" containerID="b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a" Oct 30 01:19:11.300396 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:19:11.300375 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a\": container with ID starting with b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a not found: ID does not exist" containerID="b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a" Oct 30 01:19:11.300499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.300404 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a"} err="failed to get container status \"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a\": rpc error: code = NotFound desc = could not find container \"b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a\": container with ID starting with b47294f740b1abe5f2274d36787f92f9a019e3d57b15b23905dda2ee524a064a not found: ID does not exist" Oct 30 01:19:11.311210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.311181 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\") pod \"9b15fc60-daef-4fe3-8f63-016a7011a891\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " Oct 30 01:19:11.311339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.311239 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\" (UniqueName: \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\") pod \"9b15fc60-daef-4fe3-8f63-016a7011a891\" (UID: \"9b15fc60-daef-4fe3-8f63-016a7011a891\") " Oct 30 01:19:11.312065 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.312030 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume" pod "9b15fc60-daef-4fe3-8f63-016a7011a891" (UID: "9b15fc60-daef-4fe3-8f63-016a7011a891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:11.312384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.312353 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b15fc60-daef-4fe3-8f63-016a7011a891-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.314646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.314617 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz" pod "9b15fc60-daef-4fe3-8f63-016a7011a891" (UID: "9b15fc60-daef-4fe3-8f63-016a7011a891"). InnerVolumeSpecName "kube-api-access-v2rnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:11.414186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.414156 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rnz\" (UniqueName: \"kubernetes.io/projected/9b15fc60-daef-4fe3-8f63-016a7011a891-kube-api-access-v2rnz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.503751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.503647 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.632900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.632852 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:19:11.636769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.636746 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-6f9854fb6-l79cg"] Oct 30 01:19:11.878050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:11.878019 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b15fc60-daef-4fe3-8f63-016a7011a891" path="/var/lib/kubelet/pods/9b15fc60-daef-4fe3-8f63-016a7011a891/volumes" Oct 30 01:19:13.214758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:13.214728 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:13.382274 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:13.382240 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:13.879026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:13.878989 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:14.024913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:14.024877 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:14.986647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:14.986604 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:16.661096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:16.661060 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:17.862518 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:17.862482 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.009469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.009440 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.216327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.216249 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:19:18.251405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.251374 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:19:18.399242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.399206 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.550158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.550079 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.690480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:18.690450 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:19.534136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.534106 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:19.664614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.664586 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:19:19.665292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.665271 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9b15fc60-daef-4fe3-8f63-016a7011a891" containerName="payload-processing" Oct 30 01:19:19.665292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.665287 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b15fc60-daef-4fe3-8f63-016a7011a891" containerName="payload-processing" Oct 30 01:19:19.665505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.665432 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b15fc60-daef-4fe3-8f63-016a7011a891" containerName="payload-processing" Oct 30 01:19:19.670307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.670288 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.686720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.686687 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:19:19.687824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.687800 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:19.740337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.740309 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1e8c0e-dockercfg-k7qm8\"" Oct 30 01:19:19.740670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.740309 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1e8c0e\"" Oct 30 01:19:19.839942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.836943 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwmj\" (UniqueName: \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.839942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.837109 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.938470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.938438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwmj\" (UniqueName: \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.938654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.938550 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.939412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.939386 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.961059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.961025 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwmj\" (UniqueName: \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:19.988687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:19.988659 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:20.289189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:20.289156 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:19:20.290503 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:19:20.290478 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930e38b8_c57b_4006_9cb3_4d24651285ad.slice/crio-d6f4140b08e909de15bdb1a66297c6efa7f3d8aba62300f9eaba0988e5257ca7 WatchSource:0}: Error finding container d6f4140b08e909de15bdb1a66297c6efa7f3d8aba62300f9eaba0988e5257ca7: Status 404 returned error can't find the container with id d6f4140b08e909de15bdb1a66297c6efa7f3d8aba62300f9eaba0988e5257ca7 Oct 30 01:19:20.374771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:20.374748 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" event={"ID":"930e38b8-c57b-4006-9cb3-4d24651285ad","Type":"ContainerStarted","Data":"d6f4140b08e909de15bdb1a66297c6efa7f3d8aba62300f9eaba0988e5257ca7"} Oct 30 01:19:20.827890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:20.815332 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:21.384293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:21.384257 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" event={"ID":"930e38b8-c57b-4006-9cb3-4d24651285ad","Type":"ContainerStarted","Data":"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5"} Oct 30 01:19:21.384481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:21.384362 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:21.426765 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:21.426717 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" podStartSLOduration=2.426704115 podStartE2EDuration="2.426704115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:21.42381754 +0000 UTC m=+2813.664052679" watchObservedRunningTime="2026-10-30 01:19:21.426704115 +0000 UTC m=+2813.666939228" Oct 30 01:19:22.616584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:22.616551 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:23.216536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.216502 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" podUID="3c253e9c-ee91-4361-85bf-42be96b6ed73" containerName="payload-processing" containerID="cri-o://f6248097720a9e75ef3ae58b8b6bb12bf63cb7d09f0da87d823be26372aa72a0" gracePeriod=25 Oct 30 01:19:23.251926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.251897 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" podUID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" containerName="payload-pre-processing" containerID="cri-o://00cfac1c8bf2f13c7191aa67c60ef25d0b0f3c0edffa84a094f2b3765910b7d7" gracePeriod=25 Oct 30 01:19:23.402124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.402080 5938 generic.go:356] "Generic (PLEG): container finished" podID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" containerID="00cfac1c8bf2f13c7191aa67c60ef25d0b0f3c0edffa84a094f2b3765910b7d7" exitCode=0 Oct 30 01:19:23.402368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.402345 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" event={"ID":"3930f0d4-7c13-49f7-9deb-9fc654c9b46f","Type":"ContainerDied","Data":"00cfac1c8bf2f13c7191aa67c60ef25d0b0f3c0edffa84a094f2b3765910b7d7"} Oct 30 01:19:23.404405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.404378 5938 generic.go:356] "Generic (PLEG): container finished" podID="3c253e9c-ee91-4361-85bf-42be96b6ed73" containerID="f6248097720a9e75ef3ae58b8b6bb12bf63cb7d09f0da87d823be26372aa72a0" exitCode=0 Oct 30 01:19:23.404513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.404434 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" event={"ID":"3c253e9c-ee91-4361-85bf-42be96b6ed73","Type":"ContainerDied","Data":"f6248097720a9e75ef3ae58b8b6bb12bf63cb7d09f0da87d823be26372aa72a0"} Oct 30 01:19:23.541125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.541102 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:19:23.593577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.593551 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:19:23.688945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.688919 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\") pod \"3c253e9c-ee91-4361-85bf-42be96b6ed73\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " Oct 30 01:19:23.689373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689014 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\" (UniqueName: \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\") pod \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " Oct 30 01:19:23.689373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689048 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\" (UniqueName: \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\") pod \"3c253e9c-ee91-4361-85bf-42be96b6ed73\" (UID: \"3c253e9c-ee91-4361-85bf-42be96b6ed73\") " Oct 30 01:19:23.689373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689198 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\") pod \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\" (UID: \"3930f0d4-7c13-49f7-9deb-9fc654c9b46f\") " Oct 30 01:19:23.689373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689354 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume" pod "3c253e9c-ee91-4361-85bf-42be96b6ed73" (UID: "3c253e9c-ee91-4361-85bf-42be96b6ed73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:23.689632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689602 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume" pod "3930f0d4-7c13-49f7-9deb-9fc654c9b46f" (UID: "3930f0d4-7c13-49f7-9deb-9fc654c9b46f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:23.689892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689846 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c253e9c-ee91-4361-85bf-42be96b6ed73-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:23.689892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.689883 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:23.691232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.691212 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn" pod "3930f0d4-7c13-49f7-9deb-9fc654c9b46f" (UID: "3930f0d4-7c13-49f7-9deb-9fc654c9b46f"). InnerVolumeSpecName "kube-api-access-whnfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:23.691293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.691250 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc" pod "3c253e9c-ee91-4361-85bf-42be96b6ed73" (UID: "3c253e9c-ee91-4361-85bf-42be96b6ed73"). InnerVolumeSpecName "kube-api-access-t77qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:23.790825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.790768 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnfn\" (UniqueName: \"kubernetes.io/projected/3930f0d4-7c13-49f7-9deb-9fc654c9b46f-kube-api-access-whnfn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:23.790825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:23.790793 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77qc\" (UniqueName: \"kubernetes.io/projected/3c253e9c-ee91-4361-85bf-42be96b6ed73-kube-api-access-t77qc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:24.258630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.258603 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:24.412819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.412784 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" Oct 30 01:19:24.413055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.412818 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5" event={"ID":"3c253e9c-ee91-4361-85bf-42be96b6ed73","Type":"ContainerDied","Data":"094fba61c76ae4a2b3273a8cb84f4d60c6b86216688930d6c0231062a05369e1"} Oct 30 01:19:24.413055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.412882 5938 scope.go:122] "RemoveContainer" containerID="f6248097720a9e75ef3ae58b8b6bb12bf63cb7d09f0da87d823be26372aa72a0" Oct 30 01:19:24.414712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.414683 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" event={"ID":"3930f0d4-7c13-49f7-9deb-9fc654c9b46f","Type":"ContainerDied","Data":"5de2b375e089522ee7a629c12fad08753b7e9a4c1dc7c5554ca41ec4543cc2b7"} Oct 30 01:19:24.414834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.414694 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk" Oct 30 01:19:24.424155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.424135 5938 scope.go:122] "RemoveContainer" containerID="00cfac1c8bf2f13c7191aa67c60ef25d0b0f3c0edffa84a094f2b3765910b7d7" Oct 30 01:19:24.463320 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.463298 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:19:24.474453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.474432 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-f89465fcf-zc9dk"] Oct 30 01:19:24.507351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.507300 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:19:24.515025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:24.514997 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-55569b797b-xbxw5"] Oct 30 01:19:25.877959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:25.877928 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" path="/var/lib/kubelet/pods/3930f0d4-7c13-49f7-9deb-9fc654c9b46f/volumes" Oct 30 01:19:25.878418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:25.878402 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c253e9c-ee91-4361-85bf-42be96b6ed73" path="/var/lib/kubelet/pods/3c253e9c-ee91-4361-85bf-42be96b6ed73/volumes" Oct 30 01:19:25.952494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:25.947659 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:28.187650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:28.187611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:28.836893 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:19:28.836845 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:19:29.906200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:29.906135 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:31.559032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:31.558963 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:32.394924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:32.394896 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:19:33.394101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:33.394052 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:35.048307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:35.048268 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.077962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.077917 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.144188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.144158 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:19:37.144464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.144410 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" podUID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" containerName="payload-processing" containerID="cri-o://3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37" gracePeriod=30 Oct 30 01:19:37.169592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.169564 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:19:37.169817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.169794 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" podUID="0a662095-fe89-48de-b71c-50df497a05dd" containerName="payload-pre-processing" containerID="cri-o://128bb3cf8a2aff61b5f20ba11f69768ce5116a48493991aebd9e1e1fb261c789" gracePeriod=30 Oct 30 01:19:37.265019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.264988 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.405907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.404392 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.602856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.598880 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:19:37.606832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.605759 5938 generic.go:356] "Generic (PLEG): container finished" podID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" containerID="3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37" exitCode=0 Oct 30 01:19:37.606832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.606024 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" event={"ID":"bc520c15-f605-4f32-90dd-5e6d2a5d1e41","Type":"ContainerDied","Data":"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37"} Oct 30 01:19:37.606832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.606121 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" event={"ID":"bc520c15-f605-4f32-90dd-5e6d2a5d1e41","Type":"ContainerDied","Data":"393e980e3273d92a8e29ce2f01e5d960819e30fa0d172d4cf258a8c661dab4df"} Oct 30 01:19:37.606832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.606160 5938 scope.go:122] "RemoveContainer" containerID="3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37" Oct 30 01:19:37.606832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.606331 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv" Oct 30 01:19:37.632077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.632025 5938 generic.go:356] "Generic (PLEG): container finished" podID="0a662095-fe89-48de-b71c-50df497a05dd" containerID="128bb3cf8a2aff61b5f20ba11f69768ce5116a48493991aebd9e1e1fb261c789" exitCode=0 Oct 30 01:19:37.632232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.632113 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" event={"ID":"0a662095-fe89-48de-b71c-50df497a05dd","Type":"ContainerDied","Data":"128bb3cf8a2aff61b5f20ba11f69768ce5116a48493991aebd9e1e1fb261c789"} Oct 30 01:19:37.639613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.639586 5938 scope.go:122] "RemoveContainer" containerID="3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37" Oct 30 01:19:37.640041 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:19:37.640020 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37\": container with ID starting with 3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37 not found: ID does not exist" containerID="3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37" Oct 30 01:19:37.640135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.640052 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37"} err="failed to get container status \"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37\": rpc error: code = NotFound desc = could not find container \"3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37\": container with ID starting with 3f073c9b75eb402495da5ef5b5a7f5e95688a1c7c150e9a9680a01c82a530c37 not found: ID does not exist" Oct 30 01:19:37.644112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.644091 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:19:37.660037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.660012 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.698103 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.698073 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\") pod \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " Oct 30 01:19:37.698231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.698214 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\" (UniqueName: \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\") pod \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\" (UID: \"bc520c15-f605-4f32-90dd-5e6d2a5d1e41\") " Oct 30 01:19:37.698497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.698473 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume" pod "bc520c15-f605-4f32-90dd-5e6d2a5d1e41" (UID: "bc520c15-f605-4f32-90dd-5e6d2a5d1e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:37.698855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.698835 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.700935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.700910 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd" pod "bc520c15-f605-4f32-90dd-5e6d2a5d1e41" (UID: "bc520c15-f605-4f32-90dd-5e6d2a5d1e41"). InnerVolumeSpecName "kube-api-access-ps9fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:37.800224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.800196 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\") pod \"0a662095-fe89-48de-b71c-50df497a05dd\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " Oct 30 01:19:37.800400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.800246 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\" (UniqueName: \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\") pod \"0a662095-fe89-48de-b71c-50df497a05dd\" (UID: \"0a662095-fe89-48de-b71c-50df497a05dd\") " Oct 30 01:19:37.800561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.800539 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume" pod "0a662095-fe89-48de-b71c-50df497a05dd" (UID: "0a662095-fe89-48de-b71c-50df497a05dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:37.800961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.800938 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9fd\" (UniqueName: \"kubernetes.io/projected/bc520c15-f605-4f32-90dd-5e6d2a5d1e41-kube-api-access-ps9fd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.800961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.800958 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a662095-fe89-48de-b71c-50df497a05dd-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.802585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.802556 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6" pod "0a662095-fe89-48de-b71c-50df497a05dd" (UID: "0a662095-fe89-48de-b71c-50df497a05dd"). InnerVolumeSpecName "kube-api-access-f4zw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:37.809844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.809819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.902403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.902378 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zw6\" (UniqueName: \"kubernetes.io/projected/0a662095-fe89-48de-b71c-50df497a05dd-kube-api-access-f4zw6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:37.944650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.944626 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:19:37.949217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:37.949194 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a7ca1f-66cb4d7884-6k2gv"] Oct 30 01:19:38.367119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.367089 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:38.641480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.641453 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" event={"ID":"0a662095-fe89-48de-b71c-50df497a05dd","Type":"ContainerDied","Data":"e5a607f98489627533a1d4f9b000a8b9a868ce62f4cec27633b6d98d55de241d"} Oct 30 01:19:38.641650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.641489 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj" Oct 30 01:19:38.641650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.641508 5938 scope.go:122] "RemoveContainer" containerID="128bb3cf8a2aff61b5f20ba11f69768ce5116a48493991aebd9e1e1fb261c789" Oct 30 01:19:38.691540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.691516 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:19:38.702805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:38.702781 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a7ca1f-7dd98fb4d7-28zsj"] Oct 30 01:19:39.628052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:39.628019 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:39.889682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:39.889605 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a662095-fe89-48de-b71c-50df497a05dd" path="/var/lib/kubelet/pods/0a662095-fe89-48de-b71c-50df497a05dd/volumes" Oct 30 01:19:39.890080 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:39.890065 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" path="/var/lib/kubelet/pods/bc520c15-f605-4f32-90dd-5e6d2a5d1e41/volumes" Oct 30 01:19:41.104934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:41.104903 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:41.259680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:41.259654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:41.487368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:41.487301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.390755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.390719 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.533813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.533753 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.571735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.571692 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:19:43.571992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.571968 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" podUID="5444e104-a542-44a4-b5cc-b51537462215" containerName="payload-processing" containerID="cri-o://75b9af66a7611116c39466c0e2d2570fcd041bc04a0571da555dec7d98b401e1" gracePeriod=30 Oct 30 01:19:43.591372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.591344 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:19:43.686828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.686793 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.704811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.704773 5938 generic.go:356] "Generic (PLEG): container finished" podID="5444e104-a542-44a4-b5cc-b51537462215" containerID="75b9af66a7611116c39466c0e2d2570fcd041bc04a0571da555dec7d98b401e1" exitCode=0 Oct 30 01:19:43.704970 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.704839 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" event={"ID":"5444e104-a542-44a4-b5cc-b51537462215","Type":"ContainerDied","Data":"75b9af66a7611116c39466c0e2d2570fcd041bc04a0571da555dec7d98b401e1"} Oct 30 01:19:43.705216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.705191 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" podUID="c3809237-f3dc-47c0-b9e7-441f185b5736" containerName="payload-pre-processing" containerID="cri-o://658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38" gracePeriod=30 Oct 30 01:19:43.837925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.837893 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.875312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:43.875272 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:19:44.056365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.056284 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\" (UniqueName: \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\") pod \"5444e104-a542-44a4-b5cc-b51537462215\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " Oct 30 01:19:44.056539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.056520 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\") pod \"5444e104-a542-44a4-b5cc-b51537462215\" (UID: \"5444e104-a542-44a4-b5cc-b51537462215\") " Oct 30 01:19:44.058098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.058064 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume" pod "5444e104-a542-44a4-b5cc-b51537462215" (UID: "5444e104-a542-44a4-b5cc-b51537462215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:44.060028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.059249 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx" pod "5444e104-a542-44a4-b5cc-b51537462215" (UID: "5444e104-a542-44a4-b5cc-b51537462215"). InnerVolumeSpecName "kube-api-access-9lxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:44.072922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.072590 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.104757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.104738 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:19:44.158754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.158729 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5444e104-a542-44a4-b5cc-b51537462215-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.158754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.158755 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxxx\" (UniqueName: \"kubernetes.io/projected/5444e104-a542-44a4-b5cc-b51537462215-kube-api-access-9lxxx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.260248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.260226 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\" (UniqueName: \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\") pod \"c3809237-f3dc-47c0-b9e7-441f185b5736\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " Oct 30 01:19:44.260378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.260294 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\") pod \"c3809237-f3dc-47c0-b9e7-441f185b5736\" (UID: \"c3809237-f3dc-47c0-b9e7-441f185b5736\") " Oct 30 01:19:44.260682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.260660 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume" pod "c3809237-f3dc-47c0-b9e7-441f185b5736" (UID: "c3809237-f3dc-47c0-b9e7-441f185b5736"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:44.262296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.262271 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6" pod "c3809237-f3dc-47c0-b9e7-441f185b5736" (UID: "c3809237-f3dc-47c0-b9e7-441f185b5736"). InnerVolumeSpecName "kube-api-access-bpqf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:44.314546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.314468 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.361585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.361556 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpqf6\" (UniqueName: \"kubernetes.io/projected/c3809237-f3dc-47c0-b9e7-441f185b5736-kube-api-access-bpqf6\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.361585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.361581 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3809237-f3dc-47c0-b9e7-441f185b5736-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:19:44.477154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.477125 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.715276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.715251 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" Oct 30 01:19:44.715276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.715260 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w" event={"ID":"5444e104-a542-44a4-b5cc-b51537462215","Type":"ContainerDied","Data":"7941377fec95a7c3e8b6222fa148d23598206632817cdee7f2eef039a700e4d9"} Oct 30 01:19:44.715608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.715330 5938 scope.go:122] "RemoveContainer" containerID="75b9af66a7611116c39466c0e2d2570fcd041bc04a0571da555dec7d98b401e1" Oct 30 01:19:44.718116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.718089 5938 generic.go:356] "Generic (PLEG): container finished" podID="c3809237-f3dc-47c0-b9e7-441f185b5736" containerID="658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38" exitCode=0 Oct 30 01:19:44.718253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.718142 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" Oct 30 01:19:44.718253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.718147 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" event={"ID":"c3809237-f3dc-47c0-b9e7-441f185b5736","Type":"ContainerDied","Data":"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38"} Oct 30 01:19:44.718394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.718289 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5" event={"ID":"c3809237-f3dc-47c0-b9e7-441f185b5736","Type":"ContainerDied","Data":"3a2a45ebcd26a1d7467effea801963786cbe63e7e2a77216eee217328a544973"} Oct 30 01:19:44.728738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.728718 5938 scope.go:122] "RemoveContainer" containerID="658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38" Oct 30 01:19:44.737766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.737743 5938 scope.go:122] "RemoveContainer" containerID="658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38" Oct 30 01:19:44.738080 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:19:44.738060 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38\": container with ID starting with 658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38 not found: ID does not exist" containerID="658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38" Oct 30 01:19:44.738157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.738086 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38"} err="failed to get container status \"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38\": rpc error: code = NotFound desc = could not find container \"658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38\": container with ID starting with 658e8c643a5cf5afc125d0c93850332ce88df24411dea9b99e84324a21e86a38 not found: ID does not exist" Oct 30 01:19:44.773293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.773268 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:19:44.781557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.781539 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01215f-575ff4747f-xcmr5"] Oct 30 01:19:44.802092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.802069 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.817489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.817473 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:19:44.820779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:44.820760 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01215f-588df9f96-9rc5w"] Oct 30 01:19:45.862899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:45.862820 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:45.877339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:45.877316 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5444e104-a542-44a4-b5cc-b51537462215" path="/var/lib/kubelet/pods/5444e104-a542-44a4-b5cc-b51537462215/volumes" Oct 30 01:19:45.877786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:45.877764 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3809237-f3dc-47c0-b9e7-441f185b5736" path="/var/lib/kubelet/pods/c3809237-f3dc-47c0-b9e7-441f185b5736/volumes" Oct 30 01:19:46.037488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.037449 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.252939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.242693 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.262939 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0a662095-fe89-48de-b71c-50df497a05dd" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.262967 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a662095-fe89-48de-b71c-50df497a05dd" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.262982 5938 cpu_manager.go:414] "containerMap: removing container" podUID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.262991 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263007 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3c253e9c-ee91-4361-85bf-42be96b6ed73" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263014 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c253e9c-ee91-4361-85bf-42be96b6ed73" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263040 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263048 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263066 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5444e104-a542-44a4-b5cc-b51537462215" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263073 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5444e104-a542-44a4-b5cc-b51537462215" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263089 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c3809237-f3dc-47c0-b9e7-441f185b5736" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263096 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3809237-f3dc-47c0-b9e7-441f185b5736" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263249 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5444e104-a542-44a4-b5cc-b51537462215" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263268 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3809237-f3dc-47c0-b9e7-441f185b5736" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263291 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a662095-fe89-48de-b71c-50df497a05dd" containerName="payload-pre-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263303 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc520c15-f605-4f32-90dd-5e6d2a5d1e41" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263316 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c253e9c-ee91-4361-85bf-42be96b6ed73" containerName="payload-processing" Oct 30 01:19:46.265883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.263330 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3930f0d4-7c13-49f7-9deb-9fc654c9b46f" containerName="payload-pre-processing" Oct 30 01:19:46.296170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.296124 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.309686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.309643 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:19:46.319180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.319149 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b73232\"" Oct 30 01:19:46.350776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.350748 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b73232-dockercfg-qbxdr\"" Oct 30 01:19:46.366782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.366753 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:19:46.371275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.371258 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.392703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.392676 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:19:46.408678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.408655 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.408834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.408783 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm95w\" (UniqueName: \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.510442 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.510375 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm95w\" (UniqueName: \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.510575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.510470 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmc29\" (UniqueName: \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.510645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.510619 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.510715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.510673 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.511270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.511253 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.534979 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.534947 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm95w\" (UniqueName: \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\") pod \"payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.612085 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.612061 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmc29\" (UniqueName: \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.612179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.612171 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.612781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.612765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.637163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.637141 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmc29\" (UniqueName: \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\") pod \"payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.640097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.640078 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:46.685038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.685012 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:46.946223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:46.946200 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:19:46.947857 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:19:46.947832 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c25193_d31d_48e8_8ef7_ca855ee11ab7.slice/crio-08870a66ae523c085361a268a640f148d1f31f6415bbb8ba281448656dc72b70 WatchSource:0}: Error finding container 08870a66ae523c085361a268a640f148d1f31f6415bbb8ba281448656dc72b70: Status 404 returned error can't find the container with id 08870a66ae523c085361a268a640f148d1f31f6415bbb8ba281448656dc72b70 Oct 30 01:19:47.001684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.001659 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:19:47.003691 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:19:47.003670 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d650a5a_9925_485f_9529_2bd5d3fa3de2.slice/crio-60933842cbce7aa1ac0346a1c08b6a0dc19b30c25b75c4c719a6aaca00f916ec WatchSource:0}: Error finding container 60933842cbce7aa1ac0346a1c08b6a0dc19b30c25b75c4c719a6aaca00f916ec: Status 404 returned error can't find the container with id 60933842cbce7aa1ac0346a1c08b6a0dc19b30c25b75c4c719a6aaca00f916ec Oct 30 01:19:47.016200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.015665 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:47.744209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.744172 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" event={"ID":"94c25193-d31d-48e8-8ef7-ca855ee11ab7","Type":"ContainerStarted","Data":"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0"} Oct 30 01:19:47.744413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.744234 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" event={"ID":"94c25193-d31d-48e8-8ef7-ca855ee11ab7","Type":"ContainerStarted","Data":"08870a66ae523c085361a268a640f148d1f31f6415bbb8ba281448656dc72b70"} Oct 30 01:19:47.744413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.744262 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:47.745939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.745908 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" event={"ID":"9d650a5a-9925-485f-9529-2bd5d3fa3de2","Type":"ContainerStarted","Data":"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f"} Oct 30 01:19:47.746086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.745961 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" event={"ID":"9d650a5a-9925-485f-9529-2bd5d3fa3de2","Type":"ContainerStarted","Data":"60933842cbce7aa1ac0346a1c08b6a0dc19b30c25b75c4c719a6aaca00f916ec"} Oct 30 01:19:47.746086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.746014 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:47.831801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.831409 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" podStartSLOduration=1.831391232 podStartE2EDuration="1.831391232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:47.831063195 +0000 UTC m=+2840.071298310" watchObservedRunningTime="2026-10-30 01:19:47.831391232 +0000 UTC m=+2840.071626345" Oct 30 01:19:47.832603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:47.832552 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" podStartSLOduration=1.8325410610000001 podStartE2EDuration="1.832541061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:47.791745463 +0000 UTC m=+2840.031980612" watchObservedRunningTime="2026-10-30 01:19:47.832541061 +0000 UTC m=+2840.072776177" Oct 30 01:19:48.801205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:48.801173 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.451160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:50.451127 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.783189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:50.783104 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.921063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:50.921033 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.167101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:51.167073 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:52.897299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:52.897261 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:54.842314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:54.842279 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:56.604535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:56.604500 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:58.483115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:58.483081 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:58.754595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:58.754507 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:19:58.754595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:58.754583 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:19:58.854382 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:19:58.854342 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:19:59.759267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:59.759234 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:59.926208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:19:59.926174 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:00.132761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.132728 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:20:00.137025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.137006 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.155341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.155316 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:20:00.164901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.162524 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-16c30f\"" Oct 30 01:20:00.164901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.162548 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-16c30f-dockercfg-cn7kp\"" Oct 30 01:20:00.213789 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.213762 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtk5t\" (UniqueName: \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.213958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.213910 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.319972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.319934 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtk5t\" (UniqueName: \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.326434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.326404 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.326684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.326663 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.360178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.360145 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtk5t\" (UniqueName: \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\") pod \"payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.452559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.452484 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.754316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.754293 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:20:00.755696 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:20:00.755663 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a979162_5b4f_41c9_9038_f87754ec5838.slice/crio-990fa6919dc6e8f7b6187ecbad63ad80b2cbafd508b84402822b7460dc9c9053 WatchSource:0}: Error finding container 990fa6919dc6e8f7b6187ecbad63ad80b2cbafd508b84402822b7460dc9c9053: Status 404 returned error can't find the container with id 990fa6919dc6e8f7b6187ecbad63ad80b2cbafd508b84402822b7460dc9c9053 Oct 30 01:20:00.906674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.906635 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" event={"ID":"1a979162-5b4f-41c9-9038-f87754ec5838","Type":"ContainerStarted","Data":"71f48f21ae5e1b6a71fa16fb96613be95da11b1f503e5c0ae4660c3157f2f0d1"} Oct 30 01:20:00.907138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.906699 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" event={"ID":"1a979162-5b4f-41c9-9038-f87754ec5838","Type":"ContainerStarted","Data":"990fa6919dc6e8f7b6187ecbad63ad80b2cbafd508b84402822b7460dc9c9053"} Oct 30 01:20:00.907138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.906725 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:00.947557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.947507 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" podStartSLOduration=0.94748929 podStartE2EDuration="947.48929ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:00.945553858 +0000 UTC m=+2853.185788970" watchObservedRunningTime="2026-10-30 01:20:00.94748929 +0000 UTC m=+2853.187724403" Oct 30 01:20:00.969312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:00.969285 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:01.114118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:01.114090 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:02.691960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:02.691928 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.116841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:04.116811 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.462776 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:04.462681 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.626940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:04.626898 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.968594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:04.968561 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:06.288273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:06.288241 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:08.113926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:08.113855 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:08.290901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:08.290819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:09.757420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.757389 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:09.894146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.894118 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:20:09.898422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.898401 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:09.904092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.904072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:09.915846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.915825 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:20:09.923498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.923473 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-658c99\"" Oct 30 01:20:09.923631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:09.923500 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-658c99-dockercfg-gzz8p\"" Oct 30 01:20:10.027942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.021471 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.027942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.021813 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99dx\" (UniqueName: \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.071700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.071666 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:20:10.083750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.083718 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:20:10.084045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.084024 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.123520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.123474 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.124111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.123768 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99dx\" (UniqueName: \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.124847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.124825 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.146396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.146369 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99dx\" (UniqueName: \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\") pod \"payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.213484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.213458 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:10.225094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.225064 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.225210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.225185 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dnn\" (UniqueName: \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.326917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.326851 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.327095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.326985 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dnn\" (UniqueName: \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.327797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.327764 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.350556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.350522 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dnn\" (UniqueName: \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\") pod \"payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.402366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.402345 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:10.521587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.521559 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:20:10.525277 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:20:10.525237 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a382fd9_a3df_4aa2_9a85_50a98e9884fc.slice/crio-f6d1a80056a92050fa2cfb37d63d9f293fa2797726b48f3c0aa965af6fee7952 WatchSource:0}: Error finding container f6d1a80056a92050fa2cfb37d63d9f293fa2797726b48f3c0aa965af6fee7952: Status 404 returned error can't find the container with id f6d1a80056a92050fa2cfb37d63d9f293fa2797726b48f3c0aa965af6fee7952 Oct 30 01:20:10.540916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.540889 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.673729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.673704 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:10.704387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:10.704364 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:20:10.704943 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:20:10.704921 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda896f758_01dc_4226_8698_633057b0dd24.slice/crio-680c42af5c0346153330d89da501c6a5612756fc8ff6fc69f9f1c1a6442da538 WatchSource:0}: Error finding container 680c42af5c0346153330d89da501c6a5612756fc8ff6fc69f9f1c1a6442da538: Status 404 returned error can't find the container with id 680c42af5c0346153330d89da501c6a5612756fc8ff6fc69f9f1c1a6442da538 Oct 30 01:20:11.039564 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.039530 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" event={"ID":"5a382fd9-a3df-4aa2-9a85-50a98e9884fc","Type":"ContainerStarted","Data":"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d"} Oct 30 01:20:11.039999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.039593 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:11.039999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.039616 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" event={"ID":"5a382fd9-a3df-4aa2-9a85-50a98e9884fc","Type":"ContainerStarted","Data":"f6d1a80056a92050fa2cfb37d63d9f293fa2797726b48f3c0aa965af6fee7952"} Oct 30 01:20:11.041480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.041456 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" event={"ID":"a896f758-01dc-4226-8698-633057b0dd24","Type":"ContainerStarted","Data":"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf"} Oct 30 01:20:11.041608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.041500 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" event={"ID":"a896f758-01dc-4226-8698-633057b0dd24","Type":"ContainerStarted","Data":"680c42af5c0346153330d89da501c6a5612756fc8ff6fc69f9f1c1a6442da538"} Oct 30 01:20:11.041608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.041533 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:11.080295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.080252 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" podStartSLOduration=2.08023946 podStartE2EDuration="2.08023946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:11.077445774 +0000 UTC m=+2863.317680888" watchObservedRunningTime="2026-10-30 01:20:11.08023946 +0000 UTC m=+2863.320474573" Oct 30 01:20:11.117595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.117551 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" podStartSLOduration=2.117538891 podStartE2EDuration="2.117538891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:11.115713434 +0000 UTC m=+2863.355948566" watchObservedRunningTime="2026-10-30 01:20:11.117538891 +0000 UTC m=+2863.357774042" Oct 30 01:20:11.346559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.346474 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:11.591315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.591281 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:11.849895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.843406 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:11.948131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:11.948104 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:20:12.223941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:12.223897 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:12.365768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:12.365735 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:12.506912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:12.506786 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:12.861095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:12.861068 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.331153 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:14.331116 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:15.987738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:15.987706 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.786643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:17.786605 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.055379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:19.055344 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.202174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:19.202139 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.410389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:19.410352 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.560328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:19.560294 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:20.538633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:20.538589 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:21.378155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:21.378114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:22.052859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:22.052827 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:20:22.053285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:22.053204 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:20:22.121470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:22.116204 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:23.882385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:23.882292 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:25.753008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:25.752390 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:27.259614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:27.259580 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:27.473672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:27.473639 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:27.577354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:27.577273 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:20:27.577791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:27.577759 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" containerID="cri-o://680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e" gracePeriod=30 Oct 30 01:20:27.671571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:27.671547 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:28.270654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:28.270617 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:28.835056 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:28.835022 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:20:30.339250 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:30.339216 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:20:30.370661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:30.370630 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:20:30.502041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:30.502010 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:30.642923 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:30.642887 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:30.890521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:30.890492 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:31.966731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:31.966705 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:32.781002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:32.780972 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:32.902324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:32.902300 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:20:33.008175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008146 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008256 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008306 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008456 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008462 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008517 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008578 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008598 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:33.008632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008629 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.009159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008665 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.009159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008712 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\") pod \"6c0bab2e-83ac-421d-a89b-1491900d0977\" (UID: \"6c0bab2e-83ac-421d-a89b-1491900d0977\") " Oct 30 01:20:33.009159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008710 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:33.009159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.008876 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:33.009384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009312 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:33.009929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009818 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.009929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009846 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.009929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009878 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.009929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009892 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0bab2e-83ac-421d-a89b-1491900d0977-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.009929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.009905 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.010988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.010965 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:33.011091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.010969 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:33.011091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.011018 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:20:33.011332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.011312 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth" pod "6c0bab2e-83ac-421d-a89b-1491900d0977" (UID: "6c0bab2e-83ac-421d-a89b-1491900d0977"). InnerVolumeSpecName "kube-api-access-zblth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:33.110983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.110951 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.110983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.110976 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.111160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.110989 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblth\" (UniqueName: \"kubernetes.io/projected/6c0bab2e-83ac-421d-a89b-1491900d0977-kube-api-access-zblth\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.111160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.111003 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0bab2e-83ac-421d-a89b-1491900d0977-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:33.224166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.224083 5938 generic.go:356] "Generic (PLEG): container finished" podID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerID="680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e" exitCode=0 Oct 30 01:20:33.224370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.224346 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerDied","Data":"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e"} Oct 30 01:20:33.224464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.224447 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" event={"ID":"6c0bab2e-83ac-421d-a89b-1491900d0977","Type":"ContainerDied","Data":"d328057a4e0707c9577c4fc04d9e42fb80016473eb6e44e8072145ed82853083"} Oct 30 01:20:33.224525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.224500 5938 scope.go:122] "RemoveContainer" containerID="680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e" Oct 30 01:20:33.224739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.224720 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh" Oct 30 01:20:33.241545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.241525 5938 scope.go:122] "RemoveContainer" containerID="28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680" Oct 30 01:20:33.250967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.250943 5938 scope.go:122] "RemoveContainer" containerID="680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e" Oct 30 01:20:33.251225 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:33.251204 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e\": container with ID starting with 680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e not found: ID does not exist" containerID="680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e" Oct 30 01:20:33.251285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.251231 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e"} err="failed to get container status \"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e\": rpc error: code = NotFound desc = could not find container \"680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e\": container with ID starting with 680a502302ebae152886c1d0db2d38dc0ded6ac92cd887e67934cd0ee026572e not found: ID does not exist" Oct 30 01:20:33.251285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.251249 5938 scope.go:122] "RemoveContainer" containerID="28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680" Oct 30 01:20:33.251490 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:33.251472 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680\": container with ID starting with 28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680 not found: ID does not exist" containerID="28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680" Oct 30 01:20:33.251537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.251496 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680"} err="failed to get container status \"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680\": rpc error: code = NotFound desc = could not find container \"28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680\": container with ID starting with 28ec8f1b7d2118a80bf7dfb9d766a90ca1df1023bcee02b73defa973da9c6680 not found: ID does not exist" Oct 30 01:20:33.275933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.275910 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:20:33.287113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.287088 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a7ca1f-openshift-default-77b7bf58df-t4shh"] Oct 30 01:20:33.877902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:33.877878 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" path="/var/lib/kubelet/pods/6c0bab2e-83ac-421d-a89b-1491900d0977/volumes" Oct 30 01:20:34.462461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:34.462419 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:35.339671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.339636 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" podUID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" containerName="payload-processing" containerID="cri-o://8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18" gracePeriod=25 Oct 30 01:20:35.371717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.371687 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" podUID="f69a195e-4559-41ba-9629-58f8e78bb5d7" containerName="payload-pre-processing" containerID="cri-o://8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57" gracePeriod=25 Oct 30 01:20:35.658986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.658960 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:20:35.699633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.699607 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:20:35.743804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.743769 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\") pod \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " Oct 30 01:20:35.744180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.744139 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\" (UniqueName: \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\") pod \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\" (UID: \"49a9d2ce-671c-496e-84ec-05fef4c3d6a4\") " Oct 30 01:20:35.744347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.744245 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume" pod "49a9d2ce-671c-496e-84ec-05fef4c3d6a4" (UID: "49a9d2ce-671c-496e-84ec-05fef4c3d6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:35.744794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.744760 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:35.746898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.746835 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b" pod "49a9d2ce-671c-496e-84ec-05fef4c3d6a4" (UID: "49a9d2ce-671c-496e-84ec-05fef4c3d6a4"). InnerVolumeSpecName "kube-api-access-cl77b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:35.845257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.845234 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\" (UniqueName: \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\") pod \"f69a195e-4559-41ba-9629-58f8e78bb5d7\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " Oct 30 01:20:35.845376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.845335 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\") pod \"f69a195e-4559-41ba-9629-58f8e78bb5d7\" (UID: \"f69a195e-4559-41ba-9629-58f8e78bb5d7\") " Oct 30 01:20:35.845668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.845652 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl77b\" (UniqueName: \"kubernetes.io/projected/49a9d2ce-671c-496e-84ec-05fef4c3d6a4-kube-api-access-cl77b\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:35.845750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.845676 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume" pod "f69a195e-4559-41ba-9629-58f8e78bb5d7" (UID: "f69a195e-4559-41ba-9629-58f8e78bb5d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:35.847226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.847207 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9" pod "f69a195e-4559-41ba-9629-58f8e78bb5d7" (UID: "f69a195e-4559-41ba-9629-58f8e78bb5d7"). InnerVolumeSpecName "kube-api-access-tp4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:35.947024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.947000 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f69a195e-4559-41ba-9629-58f8e78bb5d7-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:35.947024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:35.947023 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4n9\" (UniqueName: \"kubernetes.io/projected/f69a195e-4559-41ba-9629-58f8e78bb5d7-kube-api-access-tp4n9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:36.250642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.250568 5938 generic.go:356] "Generic (PLEG): container finished" podID="f69a195e-4559-41ba-9629-58f8e78bb5d7" containerID="8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57" exitCode=0 Oct 30 01:20:36.250642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.250618 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" Oct 30 01:20:36.250823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.250647 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" event={"ID":"f69a195e-4559-41ba-9629-58f8e78bb5d7","Type":"ContainerDied","Data":"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57"} Oct 30 01:20:36.250823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.250695 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977" event={"ID":"f69a195e-4559-41ba-9629-58f8e78bb5d7","Type":"ContainerDied","Data":"66358af05209f5e80d3bcbebea8f73578700025c0a28514a81be9cb55aef79ae"} Oct 30 01:20:36.250823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.250718 5938 scope.go:122] "RemoveContainer" containerID="8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57" Oct 30 01:20:36.252345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.252321 5938 generic.go:356] "Generic (PLEG): container finished" podID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" containerID="8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18" exitCode=0 Oct 30 01:20:36.252485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.252379 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" Oct 30 01:20:36.252485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.252409 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" event={"ID":"49a9d2ce-671c-496e-84ec-05fef4c3d6a4","Type":"ContainerDied","Data":"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18"} Oct 30 01:20:36.252485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.252461 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p" event={"ID":"49a9d2ce-671c-496e-84ec-05fef4c3d6a4","Type":"ContainerDied","Data":"e66c4c552d34dbc48931ce919d84def63ef19fa976283d0a85a93b6a8b328d31"} Oct 30 01:20:36.261183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.261150 5938 scope.go:122] "RemoveContainer" containerID="8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57" Oct 30 01:20:36.261469 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:36.261441 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57\": container with ID starting with 8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57 not found: ID does not exist" containerID="8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57" Oct 30 01:20:36.261541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.261473 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57"} err="failed to get container status \"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57\": rpc error: code = NotFound desc = could not find container \"8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57\": container with ID starting with 8bd45f204b9f44098722124d0383b49834046947bf6caa92a278f314894bee57 not found: ID does not exist" Oct 30 01:20:36.261541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.261490 5938 scope.go:122] "RemoveContainer" containerID="8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18" Oct 30 01:20:36.271077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.271048 5938 scope.go:122] "RemoveContainer" containerID="8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18" Oct 30 01:20:36.271357 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:36.271337 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18\": container with ID starting with 8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18 not found: ID does not exist" containerID="8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18" Oct 30 01:20:36.271443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.271364 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18"} err="failed to get container status \"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18\": rpc error: code = NotFound desc = could not find container \"8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18\": container with ID starting with 8213a98edbf3c9528a5f43203bcf752171487981251b20524061ada1d0043d18 not found: ID does not exist" Oct 30 01:20:36.302681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.302657 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:20:36.312652 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.312630 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-93079c-7d66db76d7-ls977"] Oct 30 01:20:36.345216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.345191 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:20:36.351454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.351433 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93079c-668d8c594d-qtx2p"] Oct 30 01:20:36.885730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:36.885695 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:37.877304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:37.877272 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" path="/var/lib/kubelet/pods/49a9d2ce-671c-496e-84ec-05fef4c3d6a4/volumes" Oct 30 01:20:37.877737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:37.877720 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69a195e-4559-41ba-9629-58f8e78bb5d7" path="/var/lib/kubelet/pods/f69a195e-4559-41ba-9629-58f8e78bb5d7/volumes" Oct 30 01:20:38.579716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:38.579687 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:40.217454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:40.217376 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:41.241758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:41.241726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:42.120359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:42.120316 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:44.227091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:44.227053 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:46.019840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:46.019810 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:47.877843 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:47.877810 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:49.499718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:49.499687 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:50.038430 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:50.038393 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:20:50.175826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:50.175792 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:50.320691 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:50.320611 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:50.464690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:50.464655 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:51.340805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:51.340771 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:51.489682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:51.489650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:51.634983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:51.634952 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:52.754509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:52.754474 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:20:52.755016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:52.754702 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" podUID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" containerName="maas-api" containerID="cri-o://4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d" gracePeriod=30 Oct 30 01:20:52.855294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:52.855260 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:53.070686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.070656 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:20:53.160610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.160577 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\") pod \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " Oct 30 01:20:53.160824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.160623 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\") pod \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " Oct 30 01:20:53.160824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.160784 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\") pod \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " Oct 30 01:20:53.160824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.160815 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\" (UniqueName: \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\") pod \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\" (UID: \"1bb8ae34-74d6-4df6-b2e5-7180e1042032\") " Oct 30 01:20:53.161236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.161188 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca" pod "1bb8ae34-74d6-4df6-b2e5-7180e1042032" (UID: "1bb8ae34-74d6-4df6-b2e5-7180e1042032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:53.161520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.161487 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb8ae34-74d6-4df6-b2e5-7180e1042032-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:53.163032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.163008 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm" pod "1bb8ae34-74d6-4df6-b2e5-7180e1042032" (UID: "1bb8ae34-74d6-4df6-b2e5-7180e1042032"). InnerVolumeSpecName "kube-api-access-mhhqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:53.163327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.163308 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs" pod "1bb8ae34-74d6-4df6-b2e5-7180e1042032" (UID: "1bb8ae34-74d6-4df6-b2e5-7180e1042032"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:53.163426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.163400 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls" pod "1bb8ae34-74d6-4df6-b2e5-7180e1042032" (UID: "1bb8ae34-74d6-4df6-b2e5-7180e1042032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:53.308942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.307921 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhqm\" (UniqueName: \"kubernetes.io/projected/1bb8ae34-74d6-4df6-b2e5-7180e1042032-kube-api-access-mhhqm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:53.308942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.307959 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:53.308942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.307974 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bb8ae34-74d6-4df6-b2e5-7180e1042032-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:53.403775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.403739 5938 generic.go:356] "Generic (PLEG): container finished" podID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" containerID="4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d" exitCode=0 Oct 30 01:20:53.403995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.403783 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" Oct 30 01:20:53.403995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.403827 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" event={"ID":"1bb8ae34-74d6-4df6-b2e5-7180e1042032","Type":"ContainerDied","Data":"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d"} Oct 30 01:20:53.403995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.403928 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx" event={"ID":"1bb8ae34-74d6-4df6-b2e5-7180e1042032","Type":"ContainerDied","Data":"b27c3b70943b860aeb1935f6405014c45be8f148846fe2e44f5001a159a8e764"} Oct 30 01:20:53.403995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.403961 5938 scope.go:122] "RemoveContainer" containerID="4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d" Oct 30 01:20:53.415033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.415014 5938 scope.go:122] "RemoveContainer" containerID="4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d" Oct 30 01:20:53.415301 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:53.415282 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d\": container with ID starting with 4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d not found: ID does not exist" containerID="4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d" Oct 30 01:20:53.415352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.415309 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d"} err="failed to get container status \"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d\": rpc error: code = NotFound desc = could not find container \"4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d\": container with ID starting with 4cf16aa083214ee940a027663d26f32145ee0d5c0275809ba34942021147317d not found: ID does not exist" Oct 30 01:20:53.417472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.417454 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:53.456528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.456504 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:20:53.467717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.467691 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-93079c-6fb5d56b5c-mrmnx"] Oct 30 01:20:53.563645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.563567 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:53.877741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:53.877697 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" path="/var/lib/kubelet/pods/1bb8ae34-74d6-4df6-b2e5-7180e1042032/volumes" Oct 30 01:20:54.096651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:54.096621 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.039102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.039062 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" podUID="930e38b8-c57b-4006-9cb3-4d24651285ad" containerName="payload-pre-processing" containerID="cri-o://359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5" gracePeriod=25 Oct 30 01:20:55.318957 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.318931 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:20:55.329285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.329263 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\") pod \"930e38b8-c57b-4006-9cb3-4d24651285ad\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " Oct 30 01:20:55.329424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.329321 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\" (UniqueName: \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\") pod \"930e38b8-c57b-4006-9cb3-4d24651285ad\" (UID: \"930e38b8-c57b-4006-9cb3-4d24651285ad\") " Oct 30 01:20:55.329596 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.329577 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume" pod "930e38b8-c57b-4006-9cb3-4d24651285ad" (UID: "930e38b8-c57b-4006-9cb3-4d24651285ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:20:55.329961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.329943 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/930e38b8-c57b-4006-9cb3-4d24651285ad-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.331353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.331326 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj" pod "930e38b8-c57b-4006-9cb3-4d24651285ad" (UID: "930e38b8-c57b-4006-9cb3-4d24651285ad"). InnerVolumeSpecName "kube-api-access-9rwmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:55.374538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.373772 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.420688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.420652 5938 generic.go:356] "Generic (PLEG): container finished" podID="930e38b8-c57b-4006-9cb3-4d24651285ad" containerID="359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5" exitCode=0 Oct 30 01:20:55.420910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.420696 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" Oct 30 01:20:55.420910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.420723 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" event={"ID":"930e38b8-c57b-4006-9cb3-4d24651285ad","Type":"ContainerDied","Data":"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5"} Oct 30 01:20:55.420910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.420778 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2" event={"ID":"930e38b8-c57b-4006-9cb3-4d24651285ad","Type":"ContainerDied","Data":"d6f4140b08e909de15bdb1a66297c6efa7f3d8aba62300f9eaba0988e5257ca7"} Oct 30 01:20:55.420910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.420801 5938 scope.go:122] "RemoveContainer" containerID="359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5" Oct 30 01:20:55.431043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.431019 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwmj\" (UniqueName: \"kubernetes.io/projected/930e38b8-c57b-4006-9cb3-4d24651285ad-kube-api-access-9rwmj\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:20:55.432019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.432000 5938 scope.go:122] "RemoveContainer" containerID="359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5" Oct 30 01:20:55.432309 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:55.432284 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5\": container with ID starting with 359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5 not found: ID does not exist" containerID="359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5" Oct 30 01:20:55.432378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.432315 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5"} err="failed to get container status \"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5\": rpc error: code = NotFound desc = could not find container \"359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5\": container with ID starting with 359cc63ee3bbf179f1168254b26e1f05c96740fc55de22891f8bfcef88b78df5 not found: ID does not exist" Oct 30 01:20:55.472056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.472031 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:20:55.475671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.475652 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-6dcdd584d7-q8vw2"] Oct 30 01:20:55.877343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:55.877314 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930e38b8-c57b-4006-9cb3-4d24651285ad" path="/var/lib/kubelet/pods/930e38b8-c57b-4006-9cb3-4d24651285ad/volumes" Oct 30 01:20:57.092759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:57.092730 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:58.834619 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:20:58.834580 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:20:58.842341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:20:58.842318 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.526140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:00.526107 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.701886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:00.694436 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.904443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:00.904409 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:21:00.905231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:00.904640 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" containerID="cri-o://28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d" gracePeriod=30 Oct 30 01:21:00.974175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:00.974144 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.061054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.061024 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.144462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.144429 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:21:02.144635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.144615 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" podUID="9e5b3681-820c-4666-a85f-24bbfd170aa1" containerName="payload-processing" containerID="cri-o://b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05" gracePeriod=30 Oct 30 01:21:02.166032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.166008 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:21:02.166200 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.166183 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" podUID="45c5c82c-2284-428c-beec-e54ea99ab295" containerName="payload-pre-processing" containerID="cri-o://08e568f46125f6bf04dd4633225ded783b0f749a53d0f24a14ee7dc0f9296de0" gracePeriod=30 Oct 30 01:21:02.336793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.336710 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.469422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.469391 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:21:02.475659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.475632 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.486900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.486837 5938 generic.go:356] "Generic (PLEG): container finished" podID="45c5c82c-2284-428c-beec-e54ea99ab295" containerID="08e568f46125f6bf04dd4633225ded783b0f749a53d0f24a14ee7dc0f9296de0" exitCode=0 Oct 30 01:21:02.487026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.486898 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" event={"ID":"45c5c82c-2284-428c-beec-e54ea99ab295","Type":"ContainerDied","Data":"08e568f46125f6bf04dd4633225ded783b0f749a53d0f24a14ee7dc0f9296de0"} Oct 30 01:21:02.488745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.488722 5938 generic.go:356] "Generic (PLEG): container finished" podID="9e5b3681-820c-4666-a85f-24bbfd170aa1" containerID="b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05" exitCode=0 Oct 30 01:21:02.488857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.488778 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" Oct 30 01:21:02.488857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.488805 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" event={"ID":"9e5b3681-820c-4666-a85f-24bbfd170aa1","Type":"ContainerDied","Data":"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05"} Oct 30 01:21:02.488992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.488857 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct" event={"ID":"9e5b3681-820c-4666-a85f-24bbfd170aa1","Type":"ContainerDied","Data":"80f1b93780bbcc1600cfb3060f8f89aad5d1c96e3f3a54c61b324e38ed335a2d"} Oct 30 01:21:02.488992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.488904 5938 scope.go:122] "RemoveContainer" containerID="b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05" Oct 30 01:21:02.501259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.501244 5938 scope.go:122] "RemoveContainer" containerID="b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05" Oct 30 01:21:02.501512 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:02.501493 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05\": container with ID starting with b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05 not found: ID does not exist" containerID="b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05" Oct 30 01:21:02.501580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.501522 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05"} err="failed to get container status \"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05\": rpc error: code = NotFound desc = could not find container \"b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05\": container with ID starting with b2cd85b0172c737c41ea3717d2211e5b3f6bf5778957cc610b9426448139ee05 not found: ID does not exist" Oct 30 01:21:02.504135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.504118 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:21:02.527291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.527257 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\") pod \"9e5b3681-820c-4666-a85f-24bbfd170aa1\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " Oct 30 01:21:02.527418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.527334 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\" (UniqueName: \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\") pod \"9e5b3681-820c-4666-a85f-24bbfd170aa1\" (UID: \"9e5b3681-820c-4666-a85f-24bbfd170aa1\") " Oct 30 01:21:02.527709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.527680 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume" pod "9e5b3681-820c-4666-a85f-24bbfd170aa1" (UID: "9e5b3681-820c-4666-a85f-24bbfd170aa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:02.528089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.528068 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5b3681-820c-4666-a85f-24bbfd170aa1-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:02.529906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.529886 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8" pod "9e5b3681-820c-4666-a85f-24bbfd170aa1" (UID: "9e5b3681-820c-4666-a85f-24bbfd170aa1"). InnerVolumeSpecName "kube-api-access-c2nc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.629678 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\" (UniqueName: \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\") pod \"45c5c82c-2284-428c-beec-e54ea99ab295\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.630253 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\") pod \"45c5c82c-2284-428c-beec-e54ea99ab295\" (UID: \"45c5c82c-2284-428c-beec-e54ea99ab295\") " Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.630627 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume" pod "45c5c82c-2284-428c-beec-e54ea99ab295" (UID: "45c5c82c-2284-428c-beec-e54ea99ab295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.631421 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c5c82c-2284-428c-beec-e54ea99ab295-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.631456 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2nc8\" (UniqueName: \"kubernetes.io/projected/9e5b3681-820c-4666-a85f-24bbfd170aa1-kube-api-access-c2nc8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:02.637894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.633288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx" pod "45c5c82c-2284-428c-beec-e54ea99ab295" (UID: "45c5c82c-2284-428c-beec-e54ea99ab295"). InnerVolumeSpecName "kube-api-access-456rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:02.732307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.732276 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456rx\" (UniqueName: \"kubernetes.io/projected/45c5c82c-2284-428c-beec-e54ea99ab295-kube-api-access-456rx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:02.795155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.795123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.834208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.834178 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:21:02.837360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.837341 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6d5b8-76dcb85c6d-77gct"] Oct 30 01:21:02.937841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:02.937770 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:03.075722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.075690 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:03.464003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.463932 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:03.497825 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.497798 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" Oct 30 01:21:03.497990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.497837 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9" event={"ID":"45c5c82c-2284-428c-beec-e54ea99ab295","Type":"ContainerDied","Data":"78cf497285cd7560f199102509ec4f6222878cb499dce2d502876049ea4a46d2"} Oct 30 01:21:03.497990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.497938 5938 scope.go:122] "RemoveContainer" containerID="08e568f46125f6bf04dd4633225ded783b0f749a53d0f24a14ee7dc0f9296de0" Oct 30 01:21:03.547701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.547646 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:21:03.558875 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.558844 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6d5b8-86876b7f87-7gzr9"] Oct 30 01:21:03.877988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.877956 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c5c82c-2284-428c-beec-e54ea99ab295" path="/var/lib/kubelet/pods/45c5c82c-2284-428c-beec-e54ea99ab295/volumes" Oct 30 01:21:03.878436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:03.878421 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5b3681-820c-4666-a85f-24bbfd170aa1" path="/var/lib/kubelet/pods/9e5b3681-820c-4666-a85f-24bbfd170aa1/volumes" Oct 30 01:21:04.822696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:04.822624 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:06.189514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.189453 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:21:06.275355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275319 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275371 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275430 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275466 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275541 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275620 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.275814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275617 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:06.275814 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275738 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.276041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275847 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.276041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.275905 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\") pod \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\" (UID: \"d1786f89-495b-42f2-a8d3-7d9c4bbec343\") " Oct 30 01:21:06.276417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.276394 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:06.276558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.276523 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.276681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.276544 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:06.276817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.276792 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:06.276932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.276798 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:06.279381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.279360 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:06.283128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.283103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:06.283239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.283137 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:06.283414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.283397 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq" pod "d1786f89-495b-42f2-a8d3-7d9c4bbec343" (UID: "d1786f89-495b-42f2-a8d3-7d9c4bbec343"). InnerVolumeSpecName "kube-api-access-28xkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:06.378230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378207 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378230 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378239 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378248 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378257 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xkq\" (UniqueName: \"kubernetes.io/projected/d1786f89-495b-42f2-a8d3-7d9c4bbec343-kube-api-access-28xkq\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378267 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378275 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1786f89-495b-42f2-a8d3-7d9c4bbec343-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.378392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.378283 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1786f89-495b-42f2-a8d3-7d9c4bbec343-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:06.525017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.524947 5938 generic.go:356] "Generic (PLEG): container finished" podID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerID="28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d" exitCode=0 Oct 30 01:21:06.525017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.525002 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" Oct 30 01:21:06.525183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.525004 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerDied","Data":"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d"} Oct 30 01:21:06.525183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.525075 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc" event={"ID":"d1786f89-495b-42f2-a8d3-7d9c4bbec343","Type":"ContainerDied","Data":"34b7dce5b262500caead8c290b00eee132c00ac4cc3dc3bb2933e07c470958db"} Oct 30 01:21:06.525183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.525107 5938 scope.go:122] "RemoveContainer" containerID="28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d" Oct 30 01:21:06.536470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.536452 5938 scope.go:122] "RemoveContainer" containerID="88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1" Oct 30 01:21:06.545838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.545820 5938 scope.go:122] "RemoveContainer" containerID="28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d" Oct 30 01:21:06.546091 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:06.546075 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d\": container with ID starting with 28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d not found: ID does not exist" containerID="28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d" Oct 30 01:21:06.546158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.546097 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d"} err="failed to get container status \"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d\": rpc error: code = NotFound desc = could not find container \"28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d\": container with ID starting with 28ff7ca17c70e73146a3ad3bc027376fa47ef9e899e30f3ac89e938e2884dd3d not found: ID does not exist" Oct 30 01:21:06.546158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.546113 5938 scope.go:122] "RemoveContainer" containerID="88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1" Oct 30 01:21:06.546329 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:06.546311 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1\": container with ID starting with 88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1 not found: ID does not exist" containerID="88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1" Oct 30 01:21:06.546367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.546336 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1"} err="failed to get container status \"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1\": rpc error: code = NotFound desc = could not find container \"88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1\": container with ID starting with 88c38e2313e9ceda995ffbbb822868fc656db7a4f890f014a82787f95cbfb5e1 not found: ID does not exist" Oct 30 01:21:06.575549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.575522 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:21:06.587064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.587042 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-8gwxc"] Oct 30 01:21:06.791057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:06.790984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.878859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:07.878829 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" path="/var/lib/kubelet/pods/d1786f89-495b-42f2-a8d3-7d9c4bbec343/volumes" Oct 30 01:21:09.135705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:09.133975 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:09.281357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:09.281319 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:09.623149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:09.623123 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:11.674711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:11.674674 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:11.927094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:11.927010 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:21:11.955709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:11.955682 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:21:12.095662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:12.095625 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:12.244223 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:12.244139 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:13.790687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:13.790650 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.201873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.201822 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:21:15.202365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.202105 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" podUID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" containerName="maas-api" containerID="cri-o://2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42" gracePeriod=30 Oct 30 01:21:15.311644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.311616 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.367710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.367665 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:21:15.368135 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.368104 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" podUID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" containerName="payload-processing" containerID="cri-o://3200a7287f8f785b8e364c1cfd943382a6b0dc504d2b9334d55e54427b54f053" gracePeriod=30 Oct 30 01:21:15.387662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.386357 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:21:15.387662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.386602 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" podUID="a1cfd741-fd26-4d67-9724-23e47317d156" containerName="payload-pre-processing" containerID="cri-o://8821592d4b2fd833253059b98c242c1bb1e8c2d2be616f84224607c52e267b9e" gracePeriod=30 Oct 30 01:21:15.492068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.491975 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.535904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.535879 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:21:15.610895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.609183 5938 generic.go:356] "Generic (PLEG): container finished" podID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" containerID="2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42" exitCode=0 Oct 30 01:21:15.610895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.609233 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" Oct 30 01:21:15.610895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.609261 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" event={"ID":"698dcf4b-4f94-4f88-a032-7533ff5d8f1e","Type":"ContainerDied","Data":"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42"} Oct 30 01:21:15.610895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.609330 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97" event={"ID":"698dcf4b-4f94-4f88-a032-7533ff5d8f1e","Type":"ContainerDied","Data":"d4f881330e0dff9aa2e10cd26189b1da4506b2f3cd9c2d2d831ee69d8267dddc"} Oct 30 01:21:15.610895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.609359 5938 scope.go:122] "RemoveContainer" containerID="2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42" Oct 30 01:21:15.613478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.611632 5938 generic.go:356] "Generic (PLEG): container finished" podID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" containerID="3200a7287f8f785b8e364c1cfd943382a6b0dc504d2b9334d55e54427b54f053" exitCode=0 Oct 30 01:21:15.613478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.611721 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" event={"ID":"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b","Type":"ContainerDied","Data":"3200a7287f8f785b8e364c1cfd943382a6b0dc504d2b9334d55e54427b54f053"} Oct 30 01:21:15.613946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.613922 5938 generic.go:356] "Generic (PLEG): container finished" podID="a1cfd741-fd26-4d67-9724-23e47317d156" containerID="8821592d4b2fd833253059b98c242c1bb1e8c2d2be616f84224607c52e267b9e" exitCode=0 Oct 30 01:21:15.614037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.614008 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" event={"ID":"a1cfd741-fd26-4d67-9724-23e47317d156","Type":"ContainerDied","Data":"8821592d4b2fd833253059b98c242c1bb1e8c2d2be616f84224607c52e267b9e"} Oct 30 01:21:15.621787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.621763 5938 scope.go:122] "RemoveContainer" containerID="2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42" Oct 30 01:21:15.622110 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:15.622085 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42\": container with ID starting with 2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42 not found: ID does not exist" containerID="2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42" Oct 30 01:21:15.622213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.622121 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42"} err="failed to get container status \"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42\": rpc error: code = NotFound desc = could not find container \"2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42\": container with ID starting with 2b086576f260914fab5cce6f1629898a982535f12106e1778aa446dc28d5cc42 not found: ID does not exist" Oct 30 01:21:15.633543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.633514 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\" (UniqueName: \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\") pod \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " Oct 30 01:21:15.633684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.633591 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\") pod \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " Oct 30 01:21:15.634033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.634002 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca" pod "698dcf4b-4f94-4f88-a032-7533ff5d8f1e" (UID: "698dcf4b-4f94-4f88-a032-7533ff5d8f1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:15.634229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.633647 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\") pod \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " Oct 30 01:21:15.634635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.634617 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\") pod \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\" (UID: \"698dcf4b-4f94-4f88-a032-7533ff5d8f1e\") " Oct 30 01:21:15.635712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.635690 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.637218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.637194 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs" pod "698dcf4b-4f94-4f88-a032-7533ff5d8f1e" (UID: "698dcf4b-4f94-4f88-a032-7533ff5d8f1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:15.638419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.638385 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls" pod "698dcf4b-4f94-4f88-a032-7533ff5d8f1e" (UID: "698dcf4b-4f94-4f88-a032-7533ff5d8f1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:15.639012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.638988 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q" pod "698dcf4b-4f94-4f88-a032-7533ff5d8f1e" (UID: "698dcf4b-4f94-4f88-a032-7533ff5d8f1e"). InnerVolumeSpecName "kube-api-access-q9q4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:15.644800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.644774 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:15.671006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.670983 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:21:15.737897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.737340 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9q4q\" (UniqueName: \"kubernetes.io/projected/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-kube-api-access-q9q4q\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.737897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.737385 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.737897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.737403 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/698dcf4b-4f94-4f88-a032-7533ff5d8f1e-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.805102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.804661 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:21:15.838587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.838557 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\") pod \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " Oct 30 01:21:15.838808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.838659 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\") pod \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\" (UID: \"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b\") " Oct 30 01:21:15.839081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.839051 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume" pod "0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" (UID: "0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:15.841031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.841003 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb" pod "0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" (UID: "0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b"). InnerVolumeSpecName "kube-api-access-fbxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:15.940122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940092 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\") pod \"a1cfd741-fd26-4d67-9724-23e47317d156\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " Oct 30 01:21:15.940353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940336 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\" (UniqueName: \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\") pod \"a1cfd741-fd26-4d67-9724-23e47317d156\" (UID: \"a1cfd741-fd26-4d67-9724-23e47317d156\") " Oct 30 01:21:15.940848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940546 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume" pod "a1cfd741-fd26-4d67-9724-23e47317d156" (UID: "a1cfd741-fd26-4d67-9724-23e47317d156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:15.940848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940800 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.940848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940817 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cfd741-fd26-4d67-9724-23e47317d156-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.940848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.940832 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxjb\" (UniqueName: \"kubernetes.io/projected/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b-kube-api-access-fbxjb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:15.942501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.942479 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd" pod "a1cfd741-fd26-4d67-9724-23e47317d156" (UID: "a1cfd741-fd26-4d67-9724-23e47317d156"). InnerVolumeSpecName "kube-api-access-vjfgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:15.948498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.948475 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:21:15.954343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:15.954322 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cf4567-7c9b9dfdb7-vhj97"] Oct 30 01:21:16.035464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.035439 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.042270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.042243 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfgd\" (UniqueName: \"kubernetes.io/projected/a1cfd741-fd26-4d67-9724-23e47317d156-kube-api-access-vjfgd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:16.285889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.285841 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.435839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.435808 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.623175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.623150 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" Oct 30 01:21:16.623368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.623148 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz" event={"ID":"0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b","Type":"ContainerDied","Data":"75da44bb75c404e5d971c1ba8434b5bff47eeabe29f765b32645a44c26a320e6"} Oct 30 01:21:16.623368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.623303 5938 scope.go:122] "RemoveContainer" containerID="3200a7287f8f785b8e364c1cfd943382a6b0dc504d2b9334d55e54427b54f053" Oct 30 01:21:16.624678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.624549 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" Oct 30 01:21:16.624678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.624571 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv" event={"ID":"a1cfd741-fd26-4d67-9724-23e47317d156","Type":"ContainerDied","Data":"8ba859a1c90f98c9ee765adaa037425472e26620450b36e6ec1f654940b32222"} Oct 30 01:21:16.633904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.633852 5938 scope.go:122] "RemoveContainer" containerID="8821592d4b2fd833253059b98c242c1bb1e8c2d2be616f84224607c52e267b9e" Oct 30 01:21:16.658155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.658133 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.672956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.672930 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:21:16.676600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.676581 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cf4567-57b7896bdc-d9hwv"] Oct 30 01:21:16.708514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.708491 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:21:16.712688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.712668 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cf4567-5dbcbb6f8-lwkvz"] Oct 30 01:21:16.928005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.927923 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" containerID="cri-o://101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d" gracePeriod=25 Oct 30 01:21:16.956549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:16.956517 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" containerID="cri-o://39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4" gracePeriod=25 Oct 30 01:21:17.878798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:17.878763 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" path="/var/lib/kubelet/pods/0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b/volumes" Oct 30 01:21:17.879398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:17.879381 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" path="/var/lib/kubelet/pods/698dcf4b-4f94-4f88-a032-7533ff5d8f1e/volumes" Oct 30 01:21:17.879909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:17.879893 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1cfd741-fd26-4d67-9724-23e47317d156" path="/var/lib/kubelet/pods/a1cfd741-fd26-4d67-9724-23e47317d156/volumes" Oct 30 01:21:18.117397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:18.117369 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:19.978142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:19.978106 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.262186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.262139 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:21:21.262848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.262422 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" podUID="73ccba01-1713-4d66-804b-25500dc2f52a" containerName="maas-api" containerID="cri-o://1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa" gracePeriod=30 Oct 30 01:21:21.367248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.367214 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.454565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.447219 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:21:21.454565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.447443 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" podUID="1a979162-5b4f-41c9-9038-f87754ec5838" containerName="payload-pre-processing" containerID="cri-o://71f48f21ae5e1b6a71fa16fb96613be95da11b1f503e5c0ae4660c3157f2f0d1" gracePeriod=30 Oct 30 01:21:21.548921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.548883 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.591484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.591457 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:21:21.615497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.615465 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\") pod \"73ccba01-1713-4d66-804b-25500dc2f52a\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " Oct 30 01:21:21.615680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.615547 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\") pod \"73ccba01-1713-4d66-804b-25500dc2f52a\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " Oct 30 01:21:21.615680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.615600 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\" (UniqueName: \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\") pod \"73ccba01-1713-4d66-804b-25500dc2f52a\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " Oct 30 01:21:21.615830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.615770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\") pod \"73ccba01-1713-4d66-804b-25500dc2f52a\" (UID: \"73ccba01-1713-4d66-804b-25500dc2f52a\") " Oct 30 01:21:21.616026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.615984 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca" pod "73ccba01-1713-4d66-804b-25500dc2f52a" (UID: "73ccba01-1713-4d66-804b-25500dc2f52a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:21.616500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.616470 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ccba01-1713-4d66-804b-25500dc2f52a-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.617856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.617828 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr" pod "73ccba01-1713-4d66-804b-25500dc2f52a" (UID: "73ccba01-1713-4d66-804b-25500dc2f52a"). InnerVolumeSpecName "kube-api-access-bjfhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:21.617856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.617838 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls" pod "73ccba01-1713-4d66-804b-25500dc2f52a" (UID: "73ccba01-1713-4d66-804b-25500dc2f52a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:21.618170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.618153 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs" pod "73ccba01-1713-4d66-804b-25500dc2f52a" (UID: "73ccba01-1713-4d66-804b-25500dc2f52a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:21.673510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.673467 5938 generic.go:356] "Generic (PLEG): container finished" podID="73ccba01-1713-4d66-804b-25500dc2f52a" containerID="1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa" exitCode=0 Oct 30 01:21:21.673692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.673669 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" event={"ID":"73ccba01-1713-4d66-804b-25500dc2f52a","Type":"ContainerDied","Data":"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa"} Oct 30 01:21:21.673757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.673732 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" event={"ID":"73ccba01-1713-4d66-804b-25500dc2f52a","Type":"ContainerDied","Data":"900bf805b2601467c423fb7843a56a6a21b15d14f5b829e773b33cfbb965b4fa"} Oct 30 01:21:21.673829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.673764 5938 scope.go:122] "RemoveContainer" containerID="1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa" Oct 30 01:21:21.674050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.674031 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz" Oct 30 01:21:21.680418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.680389 5938 generic.go:356] "Generic (PLEG): container finished" podID="1a979162-5b4f-41c9-9038-f87754ec5838" containerID="71f48f21ae5e1b6a71fa16fb96613be95da11b1f503e5c0ae4660c3157f2f0d1" exitCode=0 Oct 30 01:21:21.680566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.680437 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" event={"ID":"1a979162-5b4f-41c9-9038-f87754ec5838","Type":"ContainerDied","Data":"71f48f21ae5e1b6a71fa16fb96613be95da11b1f503e5c0ae4660c3157f2f0d1"} Oct 30 01:21:21.694279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.694252 5938 scope.go:122] "RemoveContainer" containerID="1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa" Oct 30 01:21:21.695881 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:21.694553 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa\": container with ID starting with 1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa not found: ID does not exist" containerID="1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa" Oct 30 01:21:21.695881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.694588 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa"} err="failed to get container status \"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa\": rpc error: code = NotFound desc = could not find container \"1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa\": container with ID starting with 1d3638e1d00f6808daa8f7c2ad983ea4ae32428b4d44cbf6f33f4bf0b6eed7fa not found: ID does not exist" Oct 30 01:21:21.718065 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.718039 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfhr\" (UniqueName: \"kubernetes.io/projected/73ccba01-1713-4d66-804b-25500dc2f52a-kube-api-access-bjfhr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.718065 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.718066 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.718245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.718082 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ccba01-1713-4d66-804b-25500dc2f52a-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.722478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.722458 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:21:21.730296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.730274 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:21:21.733982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.733852 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.734474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.734444 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-16c30f-99bf4c57c-nxxnz"] Oct 30 01:21:21.842147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.832525 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\" (UniqueName: \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\") pod \"1a979162-5b4f-41c9-9038-f87754ec5838\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " Oct 30 01:21:21.842147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.832584 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\") pod \"1a979162-5b4f-41c9-9038-f87754ec5838\" (UID: \"1a979162-5b4f-41c9-9038-f87754ec5838\") " Oct 30 01:21:21.842147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.839786 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume" pod "1a979162-5b4f-41c9-9038-f87754ec5838" (UID: "1a979162-5b4f-41c9-9038-f87754ec5838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:21.855917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.843072 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t" pod "1a979162-5b4f-41c9-9038-f87754ec5838" (UID: "1a979162-5b4f-41c9-9038-f87754ec5838"). InnerVolumeSpecName "kube-api-access-mtk5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:21.902503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.902471 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ccba01-1713-4d66-804b-25500dc2f52a" path="/var/lib/kubelet/pods/73ccba01-1713-4d66-804b-25500dc2f52a/volumes" Oct 30 01:21:21.934643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.934615 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtk5t\" (UniqueName: \"kubernetes.io/projected/1a979162-5b4f-41c9-9038-f87754ec5838-kube-api-access-mtk5t\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.934643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.934644 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a979162-5b4f-41c9-9038-f87754ec5838-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:21.985404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:21.985377 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.151525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.151487 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.288736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.288708 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.680939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.680899 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.689602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.689574 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" Oct 30 01:21:22.689778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.689610 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs" event={"ID":"1a979162-5b4f-41c9-9038-f87754ec5838","Type":"ContainerDied","Data":"990fa6919dc6e8f7b6187ecbad63ad80b2cbafd508b84402822b7460dc9c9053"} Oct 30 01:21:22.689778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.689679 5938 scope.go:122] "RemoveContainer" containerID="71f48f21ae5e1b6a71fa16fb96613be95da11b1f503e5c0ae4660c3157f2f0d1" Oct 30 01:21:22.739263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.739238 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:21:22.750527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:22.750500 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-16c30f-7b54948c4c-2cpqs"] Oct 30 01:21:23.119388 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.119358 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.133539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.133502 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:23.262702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.262672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.371372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.371289 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:21:23.371904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.371538 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" containerID="cri-o://1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185" gracePeriod=30 Oct 30 01:21:23.442725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.442699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.586450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.586414 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.877805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.877774 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a979162-5b4f-41c9-9038-f87754ec5838" path="/var/lib/kubelet/pods/1a979162-5b4f-41c9-9038-f87754ec5838/volumes" Oct 30 01:21:23.912749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.912722 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh"] Oct 30 01:21:23.913306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913292 5938 cpu_manager.go:414] "containerMap: removing container" podUID="930e38b8-c57b-4006-9cb3-4d24651285ad" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913306 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930e38b8-c57b-4006-9cb3-4d24651285ad" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913316 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1a979162-5b4f-41c9-9038-f87754ec5838" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913321 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a979162-5b4f-41c9-9038-f87754ec5838" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913340 5938 cpu_manager.go:414] "containerMap: removing container" podUID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" containerName="maas-api" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913346 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" containerName="maas-api" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913357 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f69a195e-4559-41ba-9629-58f8e78bb5d7" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913362 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69a195e-4559-41ba-9629-58f8e78bb5d7" containerName="payload-pre-processing" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913369 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913373 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913382 5938 cpu_manager.go:414] "containerMap: removing container" podUID="73ccba01-1713-4d66-804b-25500dc2f52a" containerName="maas-api" Oct 30 01:21:23.913382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913386 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ccba01-1713-4d66-804b-25500dc2f52a" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913399 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9e5b3681-820c-4666-a85f-24bbfd170aa1" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913404 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5b3681-820c-4666-a85f-24bbfd170aa1" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913411 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a1cfd741-fd26-4d67-9724-23e47317d156" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913415 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1cfd741-fd26-4d67-9724-23e47317d156" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913430 5938 cpu_manager.go:414] "containerMap: removing container" podUID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913434 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913445 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913450 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913457 5938 cpu_manager.go:414] "containerMap: removing container" podUID="45c5c82c-2284-428c-beec-e54ea99ab295" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913462 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c5c82c-2284-428c-beec-e54ea99ab295" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913470 5938 cpu_manager.go:414] "containerMap: removing container" podUID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913475 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913482 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913487 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913507 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913511 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913590 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0612cbf4-a8f8-42fe-bf9c-492b46dd3f9b" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913598 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="930e38b8-c57b-4006-9cb3-4d24651285ad" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913603 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a979162-5b4f-41c9-9038-f87754ec5838" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913611 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="698dcf4b-4f94-4f88-a032-7533ff5d8f1e" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913618 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69a195e-4559-41ba-9629-58f8e78bb5d7" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913626 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913632 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ccba01-1713-4d66-804b-25500dc2f52a" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913639 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5b3681-820c-4666-a85f-24bbfd170aa1" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913646 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1cfd741-fd26-4d67-9724-23e47317d156" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913653 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a9d2ce-671c-496e-84ec-05fef4c3d6a4" containerName="payload-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913661 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913667 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c5c82c-2284-428c-beec-e54ea99ab295" containerName="payload-pre-processing" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913673 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb8ae34-74d6-4df6-b2e5-7180e1042032" containerName="maas-api" Oct 30 01:21:23.913729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.913679 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1786f89-495b-42f2-a8d3-7d9c4bbec343" containerName="istio-proxy" Oct 30 01:21:23.918327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.918311 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:23.923343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.923320 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.933084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.933054 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:23.933196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.933119 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:21:23.935380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.935360 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh"] Oct 30 01:21:23.991608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.991579 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrkm\" (UniqueName: \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:23.991810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.991789 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:23.991887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:23.991828 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.094807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.094767 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.095017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.094820 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.095082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.095018 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrkm\" (UniqueName: \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.096606 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.096574 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.108911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.104717 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.120401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.120360 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrkm\" (UniqueName: \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.144030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.143960 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:24.232917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.232893 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:24.411581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.411507 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:21:24.412404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.412386 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:24.412446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.412406 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:24.412500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.412492 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0bab2e-83ac-421d-a89b-1491900d0977" containerName="istio-proxy" Oct 30 01:21:24.417031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.417015 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.432478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.432453 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1e8c0e\"" Oct 30 01:21:24.435900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.435848 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:21:24.466707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.466679 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1e8c0e-dockercfg-ncxtx\"" Oct 30 01:21:24.499295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.499267 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfx7f\" (UniqueName: \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.499452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.499431 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.537951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.537925 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh"] Oct 30 01:21:24.542042 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:24.542016 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ab2858_daa9_4bd5_ba2e_5874e7df541a.slice/crio-87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d WatchSource:0}: Error finding container 87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d: Status 404 returned error can't find the container with id 87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d Oct 30 01:21:24.601196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.601160 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfx7f\" (UniqueName: \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.601317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.601287 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.601987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.601965 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.623440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.623412 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfx7f\" (UniqueName: \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\") pod \"payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.731800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.731751 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:24.744240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.744218 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" event={"ID":"d2ab2858-daa9-4bd5-ba2e-5874e7df541a","Type":"ContainerStarted","Data":"87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d"} Oct 30 01:21:24.770614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.770585 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:21:24.776916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.776891 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:24.800817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.800425 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:21:24.910504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.910468 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:24.910684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.910667 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjh4t\" (UniqueName: \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:24.963012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:24.962984 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:25.012418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.012331 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjh4t\" (UniqueName: \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.012590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.012492 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.013208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.013189 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.035578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.035553 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjh4t\" (UniqueName: \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\") pod \"payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.037714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.037692 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:21:25.039751 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:25.039721 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac3a356_3f14_49a6_9b15_066b9686107e.slice/crio-7103d5631429567b965546447badf0d65242079b908ffc44c1a3a9c8ab82740f WatchSource:0}: Error finding container 7103d5631429567b965546447badf0d65242079b908ffc44c1a3a9c8ab82740f: Status 404 returned error can't find the container with id 7103d5631429567b965546447badf0d65242079b908ffc44c1a3a9c8ab82740f Oct 30 01:21:25.100679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.100658 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.407753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.407711 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:21:25.415098 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:25.415003 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode181b342_30af_4742_b693_64ea1e1721b9.slice/crio-a4ceea497ecd9d482a4a495473b8f12910cfb113f9eebb6e60c7525482adfc1b WatchSource:0}: Error finding container a4ceea497ecd9d482a4a495473b8f12910cfb113f9eebb6e60c7525482adfc1b: Status 404 returned error can't find the container with id a4ceea497ecd9d482a4a495473b8f12910cfb113f9eebb6e60c7525482adfc1b Oct 30 01:21:25.754234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.754156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" event={"ID":"aac3a356-3f14-49a6-9b15-066b9686107e","Type":"ContainerStarted","Data":"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f"} Oct 30 01:21:25.754234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.754209 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" event={"ID":"aac3a356-3f14-49a6-9b15-066b9686107e","Type":"ContainerStarted","Data":"7103d5631429567b965546447badf0d65242079b908ffc44c1a3a9c8ab82740f"} Oct 30 01:21:25.754917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.754900 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:25.756647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.756620 5938 generic.go:356] "Generic (PLEG): container finished" podID="d2ab2858-daa9-4bd5-ba2e-5874e7df541a" containerID="7a23719242bd429b6719a3a46a4580cd4bce6fe1eb4d35cc9f35b4d38615a8ae" exitCode=0 Oct 30 01:21:25.756796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.756701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" event={"ID":"d2ab2858-daa9-4bd5-ba2e-5874e7df541a","Type":"ContainerDied","Data":"7a23719242bd429b6719a3a46a4580cd4bce6fe1eb4d35cc9f35b4d38615a8ae"} Oct 30 01:21:25.758471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.758447 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" event={"ID":"e181b342-30af-4742-b693-64ea1e1721b9","Type":"ContainerStarted","Data":"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1"} Oct 30 01:21:25.758578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.758491 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" event={"ID":"e181b342-30af-4742-b693-64ea1e1721b9","Type":"ContainerStarted","Data":"a4ceea497ecd9d482a4a495473b8f12910cfb113f9eebb6e60c7525482adfc1b"} Oct 30 01:21:25.758578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.758516 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:25.797198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.797151 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" podStartSLOduration=1.797139228 podStartE2EDuration="1.797139228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:25.79513922 +0000 UTC m=+2938.035374335" watchObservedRunningTime="2026-10-30 01:21:25.797139228 +0000 UTC m=+2938.037374341" Oct 30 01:21:25.836150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:25.836106 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" podStartSLOduration=1.836092888 podStartE2EDuration="1.836092888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:25.834567034 +0000 UTC m=+2938.074802239" watchObservedRunningTime="2026-10-30 01:21:25.836092888 +0000 UTC m=+2938.076328000" Oct 30 01:21:26.925430 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:26.925409 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:26.972921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:26.972177 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.038117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.038029 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\") pod \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " Oct 30 01:21:27.038117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.038068 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\" (UniqueName: \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\") pod \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " Oct 30 01:21:27.038424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.038190 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\") pod \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\" (UID: \"d2ab2858-daa9-4bd5-ba2e-5874e7df541a\") " Oct 30 01:21:27.038424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.038394 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca" pod "d2ab2858-daa9-4bd5-ba2e-5874e7df541a" (UID: "d2ab2858-daa9-4bd5-ba2e-5874e7df541a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:27.038804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.038785 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:27.040288 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.040260 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm" pod "d2ab2858-daa9-4bd5-ba2e-5874e7df541a" (UID: "d2ab2858-daa9-4bd5-ba2e-5874e7df541a"). InnerVolumeSpecName "kube-api-access-krrkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:27.040288 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.040276 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca" pod "d2ab2858-daa9-4bd5-ba2e-5874e7df541a" (UID: "d2ab2858-daa9-4bd5-ba2e-5874e7df541a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:27.139753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.139713 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrkm\" (UniqueName: \"kubernetes.io/projected/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-kube-api-access-krrkm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:27.139753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.139746 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab2858-daa9-4bd5-ba2e-5874e7df541a-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:27.776129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.776096 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" Oct 30 01:21:27.776343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.776274 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh" event={"ID":"d2ab2858-daa9-4bd5-ba2e-5874e7df541a","Type":"ContainerDied","Data":"87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d"} Oct 30 01:21:27.776414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:27.776345 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87fe07144c5175f649c81506ef0730c5e56e7f163c8930045b7b2101c286688d" Oct 30 01:21:28.078905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.078803 5938 patch_prober.go:29] interesting pod/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:21:28.079424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.078900 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:21:28.561642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.561616 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:21:28.653806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.653721 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.653806 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.653787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.653837 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.653900 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654036 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654139 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654156 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:28.654271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654247 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.654499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.654376 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\") pod \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\" (UID: \"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983\") " Oct 30 01:21:28.655464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.655434 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:28.655775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.655750 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:28.655988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.655957 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:28.656086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.655966 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:28.656163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.656141 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.656231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.656173 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.656231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.656189 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.656231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.656205 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.656231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.656220 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.657450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.657392 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:28.663953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.659921 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:28.663953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.660291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:28.668256 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.668235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj" pod "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" (UID: "9a7c4d4c-9f6a-44b4-a3bd-d377689b8983"). InnerVolumeSpecName "kube-api-access-frblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:28.724350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.724321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:28.770317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.764993 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.770317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.765030 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.770317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.765043 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frblj\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-kube-api-access-frblj\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.770317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.765057 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:28.826277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.825469 5938 generic.go:356] "Generic (PLEG): container finished" podID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerID="1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185" exitCode=0 Oct 30 01:21:28.826277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.825671 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerDied","Data":"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185"} Oct 30 01:21:28.826277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.825758 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" event={"ID":"9a7c4d4c-9f6a-44b4-a3bd-d377689b8983","Type":"ContainerDied","Data":"a07ddebb3f13749c45f48723a0ee5cdfbb8e822bfa6276d159a00661d8728012"} Oct 30 01:21:28.826277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.825790 5938 scope.go:122] "RemoveContainer" containerID="1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185" Oct 30 01:21:28.826277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.826004 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d" Oct 30 01:21:28.844253 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.844230 5938 scope.go:122] "RemoveContainer" containerID="9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168" Oct 30 01:21:28.854553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.854525 5938 scope.go:122] "RemoveContainer" containerID="1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185" Oct 30 01:21:28.854832 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:28.854806 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185\": container with ID starting with 1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185 not found: ID does not exist" containerID="1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185" Oct 30 01:21:28.854952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.854847 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185"} err="failed to get container status \"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185\": rpc error: code = NotFound desc = could not find container \"1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185\": container with ID starting with 1ffc94f0e831e45adf9314af06a26155e129b8f1e2f9485c8b09b8100172d185 not found: ID does not exist" Oct 30 01:21:28.854952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.854910 5938 scope.go:122] "RemoveContainer" containerID="9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168" Oct 30 01:21:28.855193 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:28.855174 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168\": container with ID starting with 9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168 not found: ID does not exist" containerID="9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168" Oct 30 01:21:28.855251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.855201 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168"} err="failed to get container status \"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168\": rpc error: code = NotFound desc = could not find container \"9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168\": container with ID starting with 9338f8def8f16276b2b8fb311ac835a346846239fe1d711f4afba63d82b5d168 not found: ID does not exist" Oct 30 01:21:28.858317 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:28.858291 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:21:28.883685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.883660 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:21:28.888815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:28.888796 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cf4567-openshift-default-66b97cc97f-bhb7d"] Oct 30 01:21:29.049827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.049736 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.185878 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.185832 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.467154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.467114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.876636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.876607 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" path="/var/lib/kubelet/pods/9a7c4d4c-9f6a-44b4-a3bd-d377689b8983/volumes" Oct 30 01:21:29.937711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.937671 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24"] Oct 30 01:21:29.938336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938321 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d2ab2858-daa9-4bd5-ba2e-5874e7df541a" containerName="revoke-keys" Oct 30 01:21:29.938392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938337 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ab2858-daa9-4bd5-ba2e-5874e7df541a" containerName="revoke-keys" Oct 30 01:21:29.938392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938375 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.938392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938382 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.938492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938401 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.938492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938408 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.938558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938491 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ab2858-daa9-4bd5-ba2e-5874e7df541a" containerName="revoke-keys" Oct 30 01:21:29.938558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938504 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.938558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.938512 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7c4d4c-9f6a-44b4-a3bd-d377689b8983" containerName="istio-proxy" Oct 30 01:21:29.943579 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.943560 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:29.959344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:29.959319 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24"] Oct 30 01:21:30.000830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.000802 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:30.001011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.000995 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:21:30.083930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.083858 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2rp\" (UniqueName: \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.084078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.083974 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.084078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.084005 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.185202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.185131 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2rp\" (UniqueName: \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.185345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.185205 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.185345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.185234 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.185821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.185794 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.187612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.187586 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.211695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.211668 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2rp\" (UniqueName: \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.267269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.267244 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:30.571560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.571535 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24"] Oct 30 01:21:30.574011 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:30.573978 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9114f486_4f85_492d_bbf0_da75660563d1.slice/crio-9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd WatchSource:0}: Error finding container 9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd: Status 404 returned error can't find the container with id 9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd Oct 30 01:21:30.692557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.692518 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:30.843510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:30.843438 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" event={"ID":"9114f486-4f85-492d-bbf0-da75660563d1","Type":"ContainerStarted","Data":"9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd"} Oct 30 01:21:31.852402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:31.852372 5938 generic.go:356] "Generic (PLEG): container finished" podID="9114f486-4f85-492d-bbf0-da75660563d1" containerID="3ea4217d8ec6e5b8a9dd785a92b5622d0961f087d13cd6aee4dc6251ec3fa033" exitCode=0 Oct 30 01:21:31.852808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:31.852421 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" event={"ID":"9114f486-4f85-492d-bbf0-da75660563d1","Type":"ContainerDied","Data":"3ea4217d8ec6e5b8a9dd785a92b5622d0961f087d13cd6aee4dc6251ec3fa033"} Oct 30 01:21:31.959796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:31.959758 5938 scope.go:122] "RemoveContainer" containerID="bcf95a7c24ef301bea71dd7b06d9e4beab5ba3ff0461bf8caf8e053b1456315a" Oct 30 01:21:32.540807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:32.539126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:33.008187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.008163 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:33.133415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.133337 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:33.140126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.140102 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\") pod \"9114f486-4f85-492d-bbf0-da75660563d1\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " Oct 30 01:21:33.140239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.140206 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\" (UniqueName: \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\") pod \"9114f486-4f85-492d-bbf0-da75660563d1\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " Oct 30 01:21:33.140297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.140277 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\") pod \"9114f486-4f85-492d-bbf0-da75660563d1\" (UID: \"9114f486-4f85-492d-bbf0-da75660563d1\") " Oct 30 01:21:33.140506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.140481 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca" pod "9114f486-4f85-492d-bbf0-da75660563d1" (UID: "9114f486-4f85-492d-bbf0-da75660563d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:33.140798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.140774 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.142358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.142330 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca" pod "9114f486-4f85-492d-bbf0-da75660563d1" (UID: "9114f486-4f85-492d-bbf0-da75660563d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:33.142452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.142365 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp" pod "9114f486-4f85-492d-bbf0-da75660563d1" (UID: "9114f486-4f85-492d-bbf0-da75660563d1"). InnerVolumeSpecName "kube-api-access-qf2rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:33.242165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.242133 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf2rp\" (UniqueName: \"kubernetes.io/projected/9114f486-4f85-492d-bbf0-da75660563d1-kube-api-access-qf2rp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.242165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.242160 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9114f486-4f85-492d-bbf0-da75660563d1-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:33.868179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.868150 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" Oct 30 01:21:33.868376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.868190 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24" event={"ID":"9114f486-4f85-492d-bbf0-da75660563d1","Type":"ContainerDied","Data":"9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd"} Oct 30 01:21:33.868376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:33.868238 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6e2a7057c085ed580d84323e2435fd2b8c8d353b5476c4030e3d140d1a7fdd" Oct 30 01:21:34.144552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:34.144466 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:34.294314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:34.294282 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:34.482699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:34.482615 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:34.623528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:34.623487 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:35.081088 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:35.081052 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:21:35.113396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:35.113366 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:21:35.255629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:35.255598 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:35.479755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:35.479698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:35.621133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:35.621094 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.113721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:36.113684 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.767633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:36.767601 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:21:37.245417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.245392 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:21:37.262005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.261985 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:21:37.299556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.299529 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\") pod \"85fb993b-8494-4f3b-8e21-5330caf854e8\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " Oct 30 01:21:37.299858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.299841 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\") pod \"283de2d6-4495-4098-8291-842b85a69436\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " Oct 30 01:21:37.300145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.300121 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\" (UniqueName: \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\") pod \"85fb993b-8494-4f3b-8e21-5330caf854e8\" (UID: \"85fb993b-8494-4f3b-8e21-5330caf854e8\") " Oct 30 01:21:37.300264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.300244 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume" pod "85fb993b-8494-4f3b-8e21-5330caf854e8" (UID: "85fb993b-8494-4f3b-8e21-5330caf854e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:37.300537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.300390 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\" (UniqueName: \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\") pod \"283de2d6-4495-4098-8291-842b85a69436\" (UID: \"283de2d6-4495-4098-8291-842b85a69436\") " Oct 30 01:21:37.300732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.300520 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume" pod "283de2d6-4495-4098-8291-842b85a69436" (UID: "283de2d6-4495-4098-8291-842b85a69436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:37.301229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.301206 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fb993b-8494-4f3b-8e21-5330caf854e8-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.301338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.301232 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/283de2d6-4495-4098-8291-842b85a69436-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.303082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.303049 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv" pod "85fb993b-8494-4f3b-8e21-5330caf854e8" (UID: "85fb993b-8494-4f3b-8e21-5330caf854e8"). InnerVolumeSpecName "kube-api-access-gpwqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:37.303280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.303249 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd" pod "283de2d6-4495-4098-8291-842b85a69436" (UID: "283de2d6-4495-4098-8291-842b85a69436"). InnerVolumeSpecName "kube-api-access-4xppd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:37.402563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.402534 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwqv\" (UniqueName: \"kubernetes.io/projected/85fb993b-8494-4f3b-8e21-5330caf854e8-kube-api-access-gpwqv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.402563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.402561 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xppd\" (UniqueName: \"kubernetes.io/projected/283de2d6-4495-4098-8291-842b85a69436-kube-api-access-4xppd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.778102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.778016 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:21:37.896682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.896652 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:37.897101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.897077 5938 generic.go:356] "Generic (PLEG): container finished" podID="283de2d6-4495-4098-8291-842b85a69436" containerID="39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4" exitCode=0 Oct 30 01:21:37.897191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.897122 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" Oct 30 01:21:37.897191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.897154 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" event={"ID":"283de2d6-4495-4098-8291-842b85a69436","Type":"ContainerDied","Data":"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4"} Oct 30 01:21:37.897268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.897204 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c" event={"ID":"283de2d6-4495-4098-8291-842b85a69436","Type":"ContainerDied","Data":"9a78036afdaa8d3a971b429ed31d54a6baefa5a9cb0a4a19043026ac6cc15132"} Oct 30 01:21:37.897268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.897227 5938 scope.go:122] "RemoveContainer" containerID="39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4" Oct 30 01:21:37.898945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.898924 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" Oct 30 01:21:37.899569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.899546 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" event={"ID":"85fb993b-8494-4f3b-8e21-5330caf854e8","Type":"ContainerDied","Data":"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d"} Oct 30 01:21:37.899654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.899552 5938 generic.go:356] "Generic (PLEG): container finished" podID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerID="101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d" exitCode=0 Oct 30 01:21:37.899709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.899698 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9" event={"ID":"85fb993b-8494-4f3b-8e21-5330caf854e8","Type":"ContainerDied","Data":"9fa8f91d233841a6f72a9660fd404a427394e3f660ed8a4ed47754479ef051e6"} Oct 30 01:21:37.906840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.906810 5938 scope.go:122] "RemoveContainer" containerID="39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4" Oct 30 01:21:37.907132 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:37.907112 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4\": container with ID starting with 39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4 not found: ID does not exist" containerID="39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4" Oct 30 01:21:37.907195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.907140 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4"} err="failed to get container status \"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4\": rpc error: code = NotFound desc = could not find container \"39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4\": container with ID starting with 39fcc60fecc1454e110d49b9cd91510ad8464799fc7145995a788fe745f27bc4 not found: ID does not exist" Oct 30 01:21:37.907195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.907155 5938 scope.go:122] "RemoveContainer" containerID="101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d" Oct 30 01:21:37.916248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.916224 5938 scope.go:122] "RemoveContainer" containerID="101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d" Oct 30 01:21:37.916523 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:37.916503 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d\": container with ID starting with 101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d not found: ID does not exist" containerID="101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d" Oct 30 01:21:37.916604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.916528 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d"} err="failed to get container status \"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d\": rpc error: code = NotFound desc = could not find container \"101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d\": container with ID starting with 101ec56cd26261c9f5cb3dfc7582bcfe68d5796447cb03ce3f8a8d0ff27eec9d not found: ID does not exist" Oct 30 01:21:37.954399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.954354 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:21:37.957405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.957376 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a5e9ab-5d6f79c8c9-5rg9c"] Oct 30 01:21:37.984826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.984798 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:21:37.991517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:37.991495 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a5e9ab-74df65b974-xdkp9"] Oct 30 01:21:39.294117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:39.294084 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:39.791743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:39.791695 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:39.878437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:39.878402 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283de2d6-4495-4098-8291-842b85a69436" path="/var/lib/kubelet/pods/283de2d6-4495-4098-8291-842b85a69436/volumes" Oct 30 01:21:39.878835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:39.878821 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" path="/var/lib/kubelet/pods/85fb993b-8494-4f3b-8e21-5330caf854e8/volumes" Oct 30 01:21:39.932986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:39.932958 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:40.082319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.082234 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" podUID="a896f758-01dc-4226-8698-633057b0dd24" containerName="payload-processing" containerID="cri-o://2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf" gracePeriod=25 Oct 30 01:21:40.114660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.114628 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" podUID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" containerName="payload-pre-processing" containerID="cri-o://80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d" gracePeriod=25 Oct 30 01:21:40.432400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.426923 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:21:40.432400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.432043 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:21:40.459697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.457728 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:40.513220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.513191 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:21:40.546027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.545963 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:21:40.558315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.558287 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\") pod \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " Oct 30 01:21:40.558565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.558548 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\" (UniqueName: \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\") pod \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\" (UID: \"5a382fd9-a3df-4aa2-9a85-50a98e9884fc\") " Oct 30 01:21:40.558625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.558605 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume" pod "5a382fd9-a3df-4aa2-9a85-50a98e9884fc" (UID: "5a382fd9-a3df-4aa2-9a85-50a98e9884fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:40.559033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.559014 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:40.560565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.560545 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx" pod "5a382fd9-a3df-4aa2-9a85-50a98e9884fc" (UID: "5a382fd9-a3df-4aa2-9a85-50a98e9884fc"). InnerVolumeSpecName "kube-api-access-c99dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:40.606666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.606619 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:40.660343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.660311 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\" (UniqueName: \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\") pod \"a896f758-01dc-4226-8698-633057b0dd24\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " Oct 30 01:21:40.660491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.660396 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\") pod \"a896f758-01dc-4226-8698-633057b0dd24\" (UID: \"a896f758-01dc-4226-8698-633057b0dd24\") " Oct 30 01:21:40.660786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.660770 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99dx\" (UniqueName: \"kubernetes.io/projected/5a382fd9-a3df-4aa2-9a85-50a98e9884fc-kube-api-access-c99dx\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:40.661129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.661108 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume" pod "a896f758-01dc-4226-8698-633057b0dd24" (UID: "a896f758-01dc-4226-8698-633057b0dd24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:40.663090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.663062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn" pod "a896f758-01dc-4226-8698-633057b0dd24" (UID: "a896f758-01dc-4226-8698-633057b0dd24"). InnerVolumeSpecName "kube-api-access-p6dnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:40.762581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.762550 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a896f758-01dc-4226-8698-633057b0dd24-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:40.762581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.762576 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6dnn\" (UniqueName: \"kubernetes.io/projected/a896f758-01dc-4226-8698-633057b0dd24-kube-api-access-p6dnn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:40.921701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.921621 5938 generic.go:356] "Generic (PLEG): container finished" podID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" containerID="80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d" exitCode=0 Oct 30 01:21:40.921701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.921663 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" Oct 30 01:21:40.921955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.921700 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" event={"ID":"5a382fd9-a3df-4aa2-9a85-50a98e9884fc","Type":"ContainerDied","Data":"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d"} Oct 30 01:21:40.921955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.921763 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr" event={"ID":"5a382fd9-a3df-4aa2-9a85-50a98e9884fc","Type":"ContainerDied","Data":"f6d1a80056a92050fa2cfb37d63d9f293fa2797726b48f3c0aa965af6fee7952"} Oct 30 01:21:40.921955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.921797 5938 scope.go:122] "RemoveContainer" containerID="80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d" Oct 30 01:21:40.923155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.923136 5938 generic.go:356] "Generic (PLEG): container finished" podID="a896f758-01dc-4226-8698-633057b0dd24" containerID="2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf" exitCode=0 Oct 30 01:21:40.923251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.923190 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" Oct 30 01:21:40.923251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.923230 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" event={"ID":"a896f758-01dc-4226-8698-633057b0dd24","Type":"ContainerDied","Data":"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf"} Oct 30 01:21:40.923368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.923292 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch" event={"ID":"a896f758-01dc-4226-8698-633057b0dd24","Type":"ContainerDied","Data":"680c42af5c0346153330d89da501c6a5612756fc8ff6fc69f9f1c1a6442da538"} Oct 30 01:21:40.934138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.934120 5938 scope.go:122] "RemoveContainer" containerID="80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d" Oct 30 01:21:40.934397 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:40.934379 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d\": container with ID starting with 80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d not found: ID does not exist" containerID="80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d" Oct 30 01:21:40.934485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.934410 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d"} err="failed to get container status \"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d\": rpc error: code = NotFound desc = could not find container \"80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d\": container with ID starting with 80748872fd195b5c21a81a9ee1c1ad70cbea1d3683202c53c4292eac8616a28d not found: ID does not exist" Oct 30 01:21:40.934485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.934433 5938 scope.go:122] "RemoveContainer" containerID="2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf" Oct 30 01:21:40.943285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.943262 5938 scope.go:122] "RemoveContainer" containerID="2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf" Oct 30 01:21:40.943537 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:40.943517 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf\": container with ID starting with 2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf not found: ID does not exist" containerID="2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf" Oct 30 01:21:40.943598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.943544 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf"} err="failed to get container status \"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf\": rpc error: code = NotFound desc = could not find container \"2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf\": container with ID starting with 2183e502800ca36249411963469378be9c59393fbac9101986e59092b4f8dcbf not found: ID does not exist" Oct 30 01:21:40.971347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.971322 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:21:40.986747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:40.986717 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-77d45498f9-ftcpr"] Oct 30 01:21:41.035885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.030848 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:21:41.038491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.038457 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-7bdc65f946-t89ch"] Oct 30 01:21:41.371720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.371685 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.745382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.745293 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.876745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.876715 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" path="/var/lib/kubelet/pods/5a382fd9-a3df-4aa2-9a85-50a98e9884fc/volumes" Oct 30 01:21:41.877163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:41.877147 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a896f758-01dc-4226-8698-633057b0dd24" path="/var/lib/kubelet/pods/a896f758-01dc-4226-8698-633057b0dd24/volumes" Oct 30 01:21:43.175464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.175428 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj"] Oct 30 01:21:43.176114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176097 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9114f486-4f85-492d-bbf0-da75660563d1" containerName="revoke-keys" Oct 30 01:21:43.176166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176116 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9114f486-4f85-492d-bbf0-da75660563d1" containerName="revoke-keys" Oct 30 01:21:43.176166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176134 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" containerName="payload-pre-processing" Oct 30 01:21:43.176166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176139 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" containerName="payload-pre-processing" Oct 30 01:21:43.176166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176155 5938 cpu_manager.go:414] "containerMap: removing container" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" Oct 30 01:21:43.176166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176161 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176195 5938 cpu_manager.go:414] "containerMap: removing container" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176200 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176216 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a896f758-01dc-4226-8698-633057b0dd24" containerName="payload-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176222 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a896f758-01dc-4226-8698-633057b0dd24" containerName="payload-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176316 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a896f758-01dc-4226-8698-633057b0dd24" containerName="payload-processing" Oct 30 01:21:43.176330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176325 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9114f486-4f85-492d-bbf0-da75660563d1" containerName="revoke-keys" Oct 30 01:21:43.176513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176333 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a382fd9-a3df-4aa2-9a85-50a98e9884fc" containerName="payload-pre-processing" Oct 30 01:21:43.176513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176343 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="283de2d6-4495-4098-8291-842b85a69436" containerName="payload-pre-processing" Oct 30 01:21:43.176513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.176357 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fb993b-8494-4f3b-8e21-5330caf854e8" containerName="payload-processing" Oct 30 01:21:43.181378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.181352 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.197141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.197119 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:43.207344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.207323 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:21:43.241608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.241584 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj"] Oct 30 01:21:43.294127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.294043 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmm8\" (UniqueName: \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.294325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.294229 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.294415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.294351 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.395307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.395275 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.395488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.395346 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.395488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.395417 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmm8\" (UniqueName: \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.396053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.396030 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.397757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.397728 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.417741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.417714 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmm8\" (UniqueName: \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.495428 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.495370 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:43.532471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.532439 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:43.804780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.804749 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj"] Oct 30 01:21:43.806746 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:43.806719 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1c57f6_762d_4421_b851_be35600fde63.slice/crio-b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d WatchSource:0}: Error finding container b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d: Status 404 returned error can't find the container with id b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d Oct 30 01:21:43.945631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:43.945596 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" event={"ID":"df1c57f6-762d-4421-b851-be35600fde63","Type":"ContainerStarted","Data":"b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d"} Oct 30 01:21:44.953666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:44.953586 5938 generic.go:356] "Generic (PLEG): container finished" podID="df1c57f6-762d-4421-b851-be35600fde63" containerID="5b4aba898309914c8378c8137d5cb76502d066fdeb151760d19116a00775e092" exitCode=0 Oct 30 01:21:44.954070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:44.953674 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" event={"ID":"df1c57f6-762d-4421-b851-be35600fde63","Type":"ContainerDied","Data":"5b4aba898309914c8378c8137d5cb76502d066fdeb151760d19116a00775e092"} Oct 30 01:21:45.427760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:45.427697 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" containerID="cri-o://d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a" gracePeriod=25 Oct 30 01:21:45.433242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:45.433212 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" containerID="cri-o://c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c" gracePeriod=25 Oct 30 01:21:45.937269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:45.937241 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.127037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.127003 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:46.228918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.228811 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\") pod \"df1c57f6-762d-4421-b851-be35600fde63\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " Oct 30 01:21:46.229095 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.228991 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\" (UniqueName: \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\") pod \"df1c57f6-762d-4421-b851-be35600fde63\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " Oct 30 01:21:46.229173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.229156 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\") pod \"df1c57f6-762d-4421-b851-be35600fde63\" (UID: \"df1c57f6-762d-4421-b851-be35600fde63\") " Oct 30 01:21:46.229496 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.229472 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca" pod "df1c57f6-762d-4421-b851-be35600fde63" (UID: "df1c57f6-762d-4421-b851-be35600fde63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:46.229669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.229655 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c57f6-762d-4421-b851-be35600fde63-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.230928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.230904 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca" pod "df1c57f6-762d-4421-b851-be35600fde63" (UID: "df1c57f6-762d-4421-b851-be35600fde63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:46.231067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.231044 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8" pod "df1c57f6-762d-4421-b851-be35600fde63" (UID: "df1c57f6-762d-4421-b851-be35600fde63"). InnerVolumeSpecName "kube-api-access-zcmm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:46.331351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.331325 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmm8\" (UniqueName: \"kubernetes.io/projected/df1c57f6-762d-4421-b851-be35600fde63-kube-api-access-zcmm8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.331351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.331350 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c57f6-762d-4421-b851-be35600fde63-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:46.711226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.711194 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.856102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.856072 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:46.974348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.974269 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" Oct 30 01:21:46.974348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.974302 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj" event={"ID":"df1c57f6-762d-4421-b851-be35600fde63","Type":"ContainerDied","Data":"b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d"} Oct 30 01:21:46.974597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.974361 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00f48fb90d13f6ad5793415a00ededef5960bb4f2b526acdbb012eecc7c902d" Oct 30 01:21:46.993758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.993727 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:21:46.996680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:46.996647 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" containerID="cri-o://693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8" gracePeriod=30 Oct 30 01:21:47.103915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.103531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.576774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.576724 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.711688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.711657 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv"] Oct 30 01:21:47.712287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.712272 5938 cpu_manager.go:414] "containerMap: removing container" podUID="df1c57f6-762d-4421-b851-be35600fde63" containerName="revoke-keys" Oct 30 01:21:47.712358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.712289 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1c57f6-762d-4421-b851-be35600fde63" containerName="revoke-keys" Oct 30 01:21:47.712397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.712383 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1c57f6-762d-4421-b851-be35600fde63" containerName="revoke-keys" Oct 30 01:21:47.715989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.715973 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.744445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.744401 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv"] Oct 30 01:21:47.809431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.809402 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:21:47.809580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.809518 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:21:47.849511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.849438 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtzh\" (UniqueName: \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.849511 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.849485 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.849739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.849719 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.923259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.923229 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:47.923394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.923301 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:47.951402 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.951370 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.951526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.951488 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtzh\" (UniqueName: \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.951589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.951527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.952231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.952208 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.953672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.953653 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:47.973791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:47.973767 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtzh\" (UniqueName: \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\") pod \"maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:48.029562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:48.029537 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:48.280756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:48.280717 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:48.329955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:48.329931 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv"] Oct 30 01:21:48.331731 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:21:48.331699 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b4f97a_ed54_43ba_a81d_faf4dae44f3d.slice/crio-8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c WatchSource:0}: Error finding container 8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c: Status 404 returned error can't find the container with id 8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c Oct 30 01:21:48.989302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:48.989272 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" event={"ID":"46b4f97a-ed54-43ba-a81d-faf4dae44f3d","Type":"ContainerStarted","Data":"419dd5d7bfb7b3f5371be9e8d1d0777049f613a40b359f3525143aa364663c9d"} Oct 30 01:21:48.989759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:48.989320 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" event={"ID":"46b4f97a-ed54-43ba-a81d-faf4dae44f3d","Type":"ContainerStarted","Data":"8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c"} Oct 30 01:21:49.029941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:49.029891 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" podStartSLOduration=2.029857576 podStartE2EDuration="2.029857576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:49.027498499 +0000 UTC m=+2961.267733611" watchObservedRunningTime="2026-10-30 01:21:49.029857576 +0000 UTC m=+2961.270092690" Oct 30 01:21:49.998506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:49.998468 5938 generic.go:356] "Generic (PLEG): container finished" podID="46b4f97a-ed54-43ba-a81d-faf4dae44f3d" containerID="419dd5d7bfb7b3f5371be9e8d1d0777049f613a40b359f3525143aa364663c9d" exitCode=0 Oct 30 01:21:49.999009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:49.998530 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" event={"ID":"46b4f97a-ed54-43ba-a81d-faf4dae44f3d","Type":"ContainerDied","Data":"419dd5d7bfb7b3f5371be9e8d1d0777049f613a40b359f3525143aa364663c9d"} Oct 30 01:21:50.208184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:50.208150 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:51.152988 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.152963 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:51.293048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.292973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\" (UniqueName: \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\") pod \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " Oct 30 01:21:51.293048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.293016 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\") pod \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " Oct 30 01:21:51.293293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.293116 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\") pod \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\" (UID: \"46b4f97a-ed54-43ba-a81d-faf4dae44f3d\") " Oct 30 01:21:51.293420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.293393 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca" pod "46b4f97a-ed54-43ba-a81d-faf4dae44f3d" (UID: "46b4f97a-ed54-43ba-a81d-faf4dae44f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:51.293848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.293826 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.295216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.295191 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh" pod "46b4f97a-ed54-43ba-a81d-faf4dae44f3d" (UID: "46b4f97a-ed54-43ba-a81d-faf4dae44f3d"). InnerVolumeSpecName "kube-api-access-zqtzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:51.295449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.295432 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca" pod "46b4f97a-ed54-43ba-a81d-faf4dae44f3d" (UID: "46b4f97a-ed54-43ba-a81d-faf4dae44f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:51.399488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.398181 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtzh\" (UniqueName: \"kubernetes.io/projected/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-kube-api-access-zqtzh\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.399488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.398219 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4f97a-ed54-43ba-a81d-faf4dae44f3d-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:51.898350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:51.898322 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.013005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.012982 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" Oct 30 01:21:52.013154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.012982 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv" event={"ID":"46b4f97a-ed54-43ba-a81d-faf4dae44f3d","Type":"ContainerDied","Data":"8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c"} Oct 30 01:21:52.013154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.013105 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c20b8303197c287dd23db3ade5f5ca0da54705adffc00ee8738a0da65d6881c" Oct 30 01:21:52.297102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.297054 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:21:52.416699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416670 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.416927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416714 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.416927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416739 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.416927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.416927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416841 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.416927 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.416886 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.417230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417011 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:52.417230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417070 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:52.417230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417140 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.417230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417169 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:21:52.417230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417183 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.417513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417246 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\") pod \"f82864e7-24c8-41a5-b703-bdeba198cbdd\" (UID: \"f82864e7-24c8-41a5-b703-bdeba198cbdd\") " Oct 30 01:21:52.417513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417409 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:52.417513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417459 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:52.417793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417771 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.417938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417794 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.417938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417808 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.417938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417820 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.417938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.417831 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82864e7-24c8-41a5-b703-bdeba198cbdd-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.419304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.419280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "kube-api-access-77lst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:52.419304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.419286 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:21:52.419586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.419560 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:52.419586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.419571 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token" pod "f82864e7-24c8-41a5-b703-bdeba198cbdd" (UID: "f82864e7-24c8-41a5-b703-bdeba198cbdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:52.519301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.519244 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77lst\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-kube-api-access-77lst\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.519301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.519268 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.519301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.519277 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.519301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.519284 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82864e7-24c8-41a5-b703-bdeba198cbdd-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:21:52.546307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.546278 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 01:21:52.560804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.560771 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.661039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.661007 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh"] Oct 30 01:21:52.664145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.664119 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-11f521-tthdh"] Oct 30 01:21:52.711802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:52.711752 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:53.021541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.021505 5938 generic.go:356] "Generic (PLEG): container finished" podID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerID="693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8" exitCode=0 Oct 30 01:21:53.021721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.021561 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" Oct 30 01:21:53.021721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.021590 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerDied","Data":"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8"} Oct 30 01:21:53.021721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.021644 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh" event={"ID":"f82864e7-24c8-41a5-b703-bdeba198cbdd","Type":"ContainerDied","Data":"7be04cb4a728548807a895666c552ca2d4bfd099760a9175f95b85f0d12f45ff"} Oct 30 01:21:53.021721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.021669 5938 scope.go:122] "RemoveContainer" containerID="693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8" Oct 30 01:21:53.033227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.033209 5938 scope.go:122] "RemoveContainer" containerID="3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c" Oct 30 01:21:53.042121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.042101 5938 scope.go:122] "RemoveContainer" containerID="693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8" Oct 30 01:21:53.042390 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:53.042370 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8\": container with ID starting with 693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8 not found: ID does not exist" containerID="693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8" Oct 30 01:21:53.042454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.042395 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8"} err="failed to get container status \"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8\": rpc error: code = NotFound desc = could not find container \"693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8\": container with ID starting with 693c3ca84f21aec8bf1d0affd8104411a1861c011ad189e0b3187f4ec875dca8 not found: ID does not exist" Oct 30 01:21:53.042454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.042412 5938 scope.go:122] "RemoveContainer" containerID="3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c" Oct 30 01:21:53.042660 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:53.042644 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c\": container with ID starting with 3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c not found: ID does not exist" containerID="3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c" Oct 30 01:21:53.042698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.042665 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c"} err="failed to get container status \"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c\": rpc error: code = NotFound desc = could not find container \"3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c\": container with ID starting with 3a1de5e01e82936819727c21c84cf4e527681f1a8a02da0b121976f0a87f4d5c not found: ID does not exist" Oct 30 01:21:53.071954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.071932 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:21:53.084650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.084630 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a5e9ab-openshift-default-8498d5b8b9-m58bh"] Oct 30 01:21:53.195709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.195663 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:53.876699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.876664 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ab2858-daa9-4bd5-ba2e-5874e7df541a" path="/var/lib/kubelet/pods/d2ab2858-daa9-4bd5-ba2e-5874e7df541a/volumes" Oct 30 01:21:53.877330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:53.877308 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" path="/var/lib/kubelet/pods/f82864e7-24c8-41a5-b703-bdeba198cbdd/volumes" Oct 30 01:21:54.503228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:54.503200 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:56.248452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:56.248405 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:57.923277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:57.923236 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:57.923277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:57.923263 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:21:58.282981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:58.282846 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.425632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:58.425595 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.599413 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:58.599325 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:58.838802 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:21:58.838767 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:21:59.475686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:59.475652 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.618833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:59.618799 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.755150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:59.755067 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 01:21:59.755326 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:59.755291 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" containerID="cri-o://1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce" gracePeriod=30 Oct 30 01:21:59.813910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:21:59.813885 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:00.301855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.301819 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48"] Oct 30 01:22:00.303482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303454 5938 cpu_manager.go:414] "containerMap: removing container" podUID="46b4f97a-ed54-43ba-a81d-faf4dae44f3d" containerName="revoke-keys" Oct 30 01:22:00.303482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303483 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b4f97a-ed54-43ba-a81d-faf4dae44f3d" containerName="revoke-keys" Oct 30 01:22:00.303684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303539 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.303684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303548 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.303684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303570 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.303684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303579 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.304005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.303987 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b4f97a-ed54-43ba-a81d-faf4dae44f3d" containerName="revoke-keys" Oct 30 01:22:00.304064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.304024 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.304064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.304041 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82864e7-24c8-41a5-b703-bdeba198cbdd" containerName="istio-proxy" Oct 30 01:22:00.313089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.313063 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.323194 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.323168 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48"] Oct 30 01:22:00.370731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.370699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:00.409728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.409690 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:00.409949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.409855 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:22:00.412704 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.412678 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.412940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.412921 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn5x\" (UniqueName: \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.413031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.412984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.513601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.513579 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn5x\" (UniqueName: \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.513947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.513625 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.513947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.513679 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.514328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.514306 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.515980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.515961 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.537931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.537904 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn5x\" (UniqueName: \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.629929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.629906 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:00.929974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:00.929946 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48"] Oct 30 01:22:00.931955 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:00.931922 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11896b14_7a0b_40c1_8a5f_188c27b7fc4f.slice/crio-7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511 WatchSource:0}: Error finding container 7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511: Status 404 returned error can't find the container with id 7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511 Oct 30 01:22:01.071539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:01.071510 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" event={"ID":"11896b14-7a0b-40c1-8a5f-188c27b7fc4f","Type":"ContainerStarted","Data":"7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511"} Oct 30 01:22:02.078961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:02.078849 5938 generic.go:356] "Generic (PLEG): container finished" podID="11896b14-7a0b-40c1-8a5f-188c27b7fc4f" containerID="16cf12f37217f0b7143fa732f0dbf03067abaa808f7e1c9146e379f270bdbf57" exitCode=0 Oct 30 01:22:02.078961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:02.078916 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" event={"ID":"11896b14-7a0b-40c1-8a5f-188c27b7fc4f","Type":"ContainerDied","Data":"16cf12f37217f0b7143fa732f0dbf03067abaa808f7e1c9146e379f270bdbf57"} Oct 30 01:22:02.296460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:02.296424 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:03.231683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.231657 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:03.242377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.242352 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\") pod \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " Oct 30 01:22:03.242504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.242414 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\" (UniqueName: \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\") pod \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " Oct 30 01:22:03.242575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.242528 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\") pod \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\" (UID: \"11896b14-7a0b-40c1-8a5f-188c27b7fc4f\") " Oct 30 01:22:03.242877 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.242839 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca" pod "11896b14-7a0b-40c1-8a5f-188c27b7fc4f" (UID: "11896b14-7a0b-40c1-8a5f-188c27b7fc4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:03.243184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.243162 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:03.244437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.244412 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca" pod "11896b14-7a0b-40c1-8a5f-188c27b7fc4f" (UID: "11896b14-7a0b-40c1-8a5f-188c27b7fc4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:03.244534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.244446 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x" pod "11896b14-7a0b-40c1-8a5f-188c27b7fc4f" (UID: "11896b14-7a0b-40c1-8a5f-188c27b7fc4f"). InnerVolumeSpecName "kube-api-access-tzn5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:03.344498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.344424 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:03.344498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:03.344453 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzn5x\" (UniqueName: \"kubernetes.io/projected/11896b14-7a0b-40c1-8a5f-188c27b7fc4f-kube-api-access-tzn5x\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:04.092780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:04.092752 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" Oct 30 01:22:04.092780 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:04.092771 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48" event={"ID":"11896b14-7a0b-40c1-8a5f-188c27b7fc4f","Type":"ContainerDied","Data":"7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511"} Oct 30 01:22:04.093011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:04.092811 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd6cbaaf32fe599f6995f0d0b8f23f6104ff36da2fa6d9287ff32b1e2041511" Oct 30 01:22:04.404467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:04.404425 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.050285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.050252 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 01:22:05.100999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.100968 5938 generic.go:356] "Generic (PLEG): container finished" podID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerID="1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce" exitCode=0 Oct 30 01:22:05.101142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.101026 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" Oct 30 01:22:05.101142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.101061 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerDied","Data":"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce"} Oct 30 01:22:05.101142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.101132 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h" event={"ID":"0933cf58-f17a-4da7-8ab9-cc48aafbe599","Type":"ContainerDied","Data":"c088e9d7a2d84a4b857f789dd0db88ba620f83e547b7be2fe015fb1a09b11cad"} Oct 30 01:22:05.101284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.101160 5938 scope.go:122] "RemoveContainer" containerID="1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce" Oct 30 01:22:05.113021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.112996 5938 scope.go:122] "RemoveContainer" containerID="0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5" Oct 30 01:22:05.126575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.126551 5938 scope.go:122] "RemoveContainer" containerID="1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce" Oct 30 01:22:05.126819 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:05.126801 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce\": container with ID starting with 1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce not found: ID does not exist" containerID="1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce" Oct 30 01:22:05.126961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.126826 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce"} err="failed to get container status \"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce\": rpc error: code = NotFound desc = could not find container \"1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce\": container with ID starting with 1ad5d084bd3361dc3ec8078fd04c7e0ebf3aa5b3af8ede96008d69a1b02a51ce not found: ID does not exist" Oct 30 01:22:05.126961 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.126841 5938 scope.go:122] "RemoveContainer" containerID="0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5" Oct 30 01:22:05.127153 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:05.127137 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5\": container with ID starting with 0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5 not found: ID does not exist" containerID="0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5" Oct 30 01:22:05.127224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.127154 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5"} err="failed to get container status \"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5\": rpc error: code = NotFound desc = could not find container \"0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5\": container with ID starting with 0cb046906283551f5e2240ca734f46ae9a0be50f824b24f5d4536c39831f04b5 not found: ID does not exist" Oct 30 01:22:05.167458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167434 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167512 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167586 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167727 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167799 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167774 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167825 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.167953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167836 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:05.167953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167941 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.168132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167984 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\") pod \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\" (UID: \"0933cf58-f17a-4da7-8ab9-cc48aafbe599\") " Oct 30 01:22:05.168132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.167988 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.168132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168074 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.168132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168090 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.168363 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168205 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.168525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168474 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.168639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168540 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.168639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168564 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.168639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168583 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.168639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.168599 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.169828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.169798 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:05.169986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.169903 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "kube-api-access-8h778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:05.169986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.169922 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.170111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.170074 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token" pod "0933cf58-f17a-4da7-8ab9-cc48aafbe599" (UID: "0933cf58-f17a-4da7-8ab9-cc48aafbe599"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:05.269962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.269941 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.269962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.269963 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h778\" (UniqueName: \"kubernetes.io/projected/0933cf58-f17a-4da7-8ab9-cc48aafbe599-kube-api-access-8h778\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.270113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.269973 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.270113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.269982 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0933cf58-f17a-4da7-8ab9-cc48aafbe599-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.447058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.447031 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 01:22:05.458154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.458130 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-7jp2h"] Oct 30 01:22:05.640903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.636484 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.679633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.679357 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:22:05.776565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.776533 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:22:05.778345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.778319 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\" (UniqueName: \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\") pod \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " Oct 30 01:22:05.778775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.778752 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\") pod \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\" (UID: \"c0197d7f-b6ee-4137-8efe-010ecea1cbf2\") " Oct 30 01:22:05.779424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.779401 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume" pod "c0197d7f-b6ee-4137-8efe-010ecea1cbf2" (UID: "c0197d7f-b6ee-4137-8efe-010ecea1cbf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:05.781209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.781184 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl" pod "c0197d7f-b6ee-4137-8efe-010ecea1cbf2" (UID: "c0197d7f-b6ee-4137-8efe-010ecea1cbf2"). InnerVolumeSpecName "kube-api-access-jqbcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:05.804013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.803991 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.877450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.877422 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" path="/var/lib/kubelet/pods/0933cf58-f17a-4da7-8ab9-cc48aafbe599/volumes" Oct 30 01:22:05.879802 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.879780 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\") pod \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " Oct 30 01:22:05.879940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.879929 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\" (UniqueName: \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\") pod \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\" (UID: \"46574bef-398f-4d7e-bbda-0eecd6ce2f31\") " Oct 30 01:22:05.880242 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.880217 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume" pod "46574bef-398f-4d7e-bbda-0eecd6ce2f31" (UID: "46574bef-398f-4d7e-bbda-0eecd6ce2f31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:05.880313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.880256 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.880313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.880276 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbcl\" (UniqueName: \"kubernetes.io/projected/c0197d7f-b6ee-4137-8efe-010ecea1cbf2-kube-api-access-jqbcl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.882113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.882090 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7" pod "46574bef-398f-4d7e-bbda-0eecd6ce2f31" (UID: "46574bef-398f-4d7e-bbda-0eecd6ce2f31"). InnerVolumeSpecName "kube-api-access-7fmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:05.991980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.989908 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46574bef-398f-4d7e-bbda-0eecd6ce2f31-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.991980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:05.989948 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmd7\" (UniqueName: \"kubernetes.io/projected/46574bef-398f-4d7e-bbda-0eecd6ce2f31-kube-api-access-7fmd7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.019624 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020504 5938 cpu_manager.go:414] "containerMap: removing container" podUID="11896b14-7a0b-40c1-8a5f-188c27b7fc4f" containerName="revoke-keys" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020524 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11896b14-7a0b-40c1-8a5f-188c27b7fc4f" containerName="revoke-keys" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020552 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020559 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020618 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020628 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020646 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020653 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020670 5938 cpu_manager.go:414] "containerMap: removing container" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020679 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020811 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020831 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerName="payload-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020844 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0933cf58-f17a-4da7-8ab9-cc48aafbe599" containerName="istio-proxy" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020854 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerName="payload-pre-processing" Oct 30 01:22:06.021108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.020882 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="11896b14-7a0b-40c1-8a5f-188c27b7fc4f" containerName="revoke-keys" Oct 30 01:22:06.028638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.028609 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.042016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.041992 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:22:06.042609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.042587 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-658c99\"" Oct 30 01:22:06.042707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.042663 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-658c99-dockercfg-7965q\"" Oct 30 01:22:06.091082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.091057 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qkgg\" (UniqueName: \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.091244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.091222 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.109575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.109547 5938 generic.go:356] "Generic (PLEG): container finished" podID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" containerID="d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a" exitCode=0 Oct 30 01:22:06.109690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.109579 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" event={"ID":"c0197d7f-b6ee-4137-8efe-010ecea1cbf2","Type":"ContainerDied","Data":"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a"} Oct 30 01:22:06.109690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.109624 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" event={"ID":"c0197d7f-b6ee-4137-8efe-010ecea1cbf2","Type":"ContainerDied","Data":"eeeabe9ddf0a1e53f7e54cb447166db16baf79d1517eb19db0892aaeccba5def"} Oct 30 01:22:06.109690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.109647 5938 scope.go:122] "RemoveContainer" containerID="d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a" Oct 30 01:22:06.109690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.109592 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh" Oct 30 01:22:06.112386 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.112365 5938 generic.go:356] "Generic (PLEG): container finished" podID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" containerID="c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c" exitCode=0 Oct 30 01:22:06.112498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.112420 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" Oct 30 01:22:06.112498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.112452 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" event={"ID":"46574bef-398f-4d7e-bbda-0eecd6ce2f31","Type":"ContainerDied","Data":"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c"} Oct 30 01:22:06.112605 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.112498 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm" event={"ID":"46574bef-398f-4d7e-bbda-0eecd6ce2f31","Type":"ContainerDied","Data":"d39e9f6ae3d81f0e74e781a639e6220287e9766b5b4ef029db8e7e340b2d34fc"} Oct 30 01:22:06.120831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.120707 5938 scope.go:122] "RemoveContainer" containerID="d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a" Oct 30 01:22:06.121058 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:06.121033 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a\": container with ID starting with d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a not found: ID does not exist" containerID="d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a" Oct 30 01:22:06.121171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.121063 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a"} err="failed to get container status \"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a\": rpc error: code = NotFound desc = could not find container \"d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a\": container with ID starting with d036d3e53df9f9721dc7c47731eaaf35df4f3d5b3834ad809ee204f1f9aea31a not found: ID does not exist" Oct 30 01:22:06.121171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.121080 5938 scope.go:122] "RemoveContainer" containerID="c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c" Oct 30 01:22:06.130059 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.130043 5938 scope.go:122] "RemoveContainer" containerID="c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c" Oct 30 01:22:06.130280 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:06.130262 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c\": container with ID starting with c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c not found: ID does not exist" containerID="c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c" Oct 30 01:22:06.130335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.130290 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c"} err="failed to get container status \"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c\": rpc error: code = NotFound desc = could not find container \"c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c\": container with ID starting with c7c57e9e8ac5f410347b4a1efefb0dc9b07740c5bb3789964f3e1a16e793b77c not found: ID does not exist" Oct 30 01:22:06.159412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.159390 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:22:06.162106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.162087 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:22:06.167353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.167334 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.168788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.168765 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-beeba4-7d8695bc7f-xqctm"] Oct 30 01:22:06.184437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.184416 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:22:06.193041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.193005 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.193199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.193178 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.193306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.193290 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wv5f\" (UniqueName: \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.193422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.193405 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qkgg\" (UniqueName: \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.194071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.194045 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.205127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.205106 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:22:06.209168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.209148 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-beeba4-c8c5c5566-rqsqh"] Oct 30 01:22:06.216215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.216187 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qkgg\" (UniqueName: \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\") pod \"payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.294294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.294238 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.294429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.294351 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wv5f\" (UniqueName: \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.294960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.294929 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.316528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.316505 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wv5f\" (UniqueName: \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\") pod \"payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.343624 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.343606 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:06.481467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.481439 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:06.645280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.645257 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:22:06.647342 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:06.647311 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5303b6_9f78_4d26_9ff3_2ff915f2422b.slice/crio-dc75f47c455627018f1e81370e95ca82b1832332527f93ffb7a11122115d6271 WatchSource:0}: Error finding container dc75f47c455627018f1e81370e95ca82b1832332527f93ffb7a11122115d6271: Status 404 returned error can't find the container with id dc75f47c455627018f1e81370e95ca82b1832332527f93ffb7a11122115d6271 Oct 30 01:22:06.783110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.783088 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:22:06.785682 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:06.785655 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671ec99d_4f3e_4ed1_aca5_01f510303092.slice/crio-b9a2aedb523858e74dd7516006688078456c7261dcd19d7882ef4a46e056e456 WatchSource:0}: Error finding container b9a2aedb523858e74dd7516006688078456c7261dcd19d7882ef4a46e056e456: Status 404 returned error can't find the container with id b9a2aedb523858e74dd7516006688078456c7261dcd19d7882ef4a46e056e456 Oct 30 01:22:06.944935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:06.944823 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:07.123166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.123133 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" event={"ID":"0c5303b6-9f78-4d26-9ff3-2ff915f2422b","Type":"ContainerStarted","Data":"6b59345b321b418c58d585dd48de26c0a13b47917d0b28adfbac0aded005aa85"} Oct 30 01:22:07.123364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.123188 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" event={"ID":"0c5303b6-9f78-4d26-9ff3-2ff915f2422b","Type":"ContainerStarted","Data":"dc75f47c455627018f1e81370e95ca82b1832332527f93ffb7a11122115d6271"} Oct 30 01:22:07.123364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.123221 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:07.134785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.134663 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" event={"ID":"671ec99d-4f3e-4ed1-aca5-01f510303092","Type":"ContainerStarted","Data":"b39a76529b1df7a503687df64620c26251a3147665c6fa1aabaf9ba872919549"} Oct 30 01:22:07.134785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.134753 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" event={"ID":"671ec99d-4f3e-4ed1-aca5-01f510303092","Type":"ContainerStarted","Data":"b9a2aedb523858e74dd7516006688078456c7261dcd19d7882ef4a46e056e456"} Oct 30 01:22:07.135245 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.135207 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:07.165296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.165241 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" podStartSLOduration=2.165224237 podStartE2EDuration="2.165224237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:07.161707484 +0000 UTC m=+2979.401942599" watchObservedRunningTime="2026-10-30 01:22:07.165224237 +0000 UTC m=+2979.405459351" Oct 30 01:22:07.204752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.204665 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" podStartSLOduration=1.20465011 podStartE2EDuration="1.20465011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:07.201517434 +0000 UTC m=+2979.441752575" watchObservedRunningTime="2026-10-30 01:22:07.20465011 +0000 UTC m=+2979.444885229" Oct 30 01:22:07.568509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.568421 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" containerID="cri-o://b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58" gracePeriod=45 Oct 30 01:22:07.843690 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.843661 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 01:22:07.878351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.878326 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46574bef-398f-4d7e-bbda-0eecd6ce2f31" path="/var/lib/kubelet/pods/46574bef-398f-4d7e-bbda-0eecd6ce2f31/volumes" Oct 30 01:22:07.878738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.878724 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0197d7f-b6ee-4137-8efe-010ecea1cbf2" path="/var/lib/kubelet/pods/c0197d7f-b6ee-4137-8efe-010ecea1cbf2/volumes" Oct 30 01:22:07.916126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.916298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916140 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.916298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916217 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\" (UniqueName: \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.916298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916274 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.916487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916365 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.916487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.916424 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\") pod \"4e777165-f922-436a-a9a5-d21765c8ff20\" (UID: \"4e777165-f922-436a-a9a5-d21765c8ff20\") " Oct 30 01:22:07.917043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.917013 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.917264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.917238 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.917449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.917419 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.919641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.919615 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "kube-api-access-58hz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:07.923483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.923458 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.923712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:07.923682 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs" pod "4e777165-f922-436a-a9a5-d21765c8ff20" (UID: "4e777165-f922-436a-a9a5-d21765c8ff20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:08.018370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018339 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e777165-f922-436a-a9a5-d21765c8ff20-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.018370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018367 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.018531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018379 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.018531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018390 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.018531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018402 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hz5\" (UniqueName: \"kubernetes.io/projected/4e777165-f922-436a-a9a5-d21765c8ff20-kube-api-access-58hz5\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.018531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.018417 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e777165-f922-436a-a9a5-d21765c8ff20-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.145769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.145737 5938 generic.go:356] "Generic (PLEG): container finished" podID="4e777165-f922-436a-a9a5-d21765c8ff20" containerID="b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58" exitCode=0 Oct 30 01:22:08.145958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.145781 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" Oct 30 01:22:08.145958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.145824 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" event={"ID":"4e777165-f922-436a-a9a5-d21765c8ff20","Type":"ContainerDied","Data":"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58"} Oct 30 01:22:08.145958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.145898 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh" event={"ID":"4e777165-f922-436a-a9a5-d21765c8ff20","Type":"ContainerDied","Data":"1bc02b6bbd9ad932cba8ba6a1f2761c6d15ad7872ffcc22b7e0fe10769b02cd7"} Oct 30 01:22:08.145958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.145913 5938 scope.go:122] "RemoveContainer" containerID="b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58" Oct 30 01:22:08.156600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.156583 5938 scope.go:122] "RemoveContainer" containerID="b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58" Oct 30 01:22:08.156881 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:08.156848 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58\": container with ID starting with b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58 not found: ID does not exist" containerID="b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58" Oct 30 01:22:08.156978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.156889 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58"} err="failed to get container status \"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58\": rpc error: code = NotFound desc = could not find container \"b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58\": container with ID starting with b7ac753aab81a7944af8e9afe0663b90031030eb688708a3c02c79b7d02feb58 not found: ID does not exist" Oct 30 01:22:08.195774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.195749 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 01:22:08.206870 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.206841 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6f4fba/premium-w5-6f4fba-kserve-c9c955b96-9r7gh"] Oct 30 01:22:08.555603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:08.555521 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:09.877075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:09.876997 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" path="/var/lib/kubelet/pods/4e777165-f922-436a-a9a5-d21765c8ff20/volumes" Oct 30 01:22:10.263269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:10.263176 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:10.873882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:10.873817 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.026712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:11.026676 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.676610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:11.676582 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.316734 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:12.316700 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.424728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:12.424698 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24"] Oct 30 01:22:12.435755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:12.435716 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-beeba4-54m24"] Oct 30 01:22:12.464003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:12.463971 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.971378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:12.971349 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:13.875915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:13.875884 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9114f486-4f85-492d-bbf0-da75660563d1" path="/var/lib/kubelet/pods/9114f486-4f85-492d-bbf0-da75660563d1/volumes" Oct 30 01:22:14.304289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:14.304182 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:15.933659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:15.933626 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.002977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.002948 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:22:16.003215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.003189 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" podUID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" containerName="payload-processing" containerID="cri-o://0795254e8effc1c21554934a9bd92f3b6586ba857af5b5f3d72783bf3fde0afe" gracePeriod=30 Oct 30 01:22:16.125179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.125148 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.205760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.205523 5938 generic.go:356] "Generic (PLEG): container finished" podID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" containerID="0795254e8effc1c21554934a9bd92f3b6586ba857af5b5f3d72783bf3fde0afe" exitCode=0 Oct 30 01:22:16.205760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.205663 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" event={"ID":"19481de2-9d24-474e-abd3-82c1bfa0b2f4","Type":"ContainerDied","Data":"0795254e8effc1c21554934a9bd92f3b6586ba857af5b5f3d72783bf3fde0afe"} Oct 30 01:22:16.264913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.264856 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.283069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.283044 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:22:16.420367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.419849 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\" (UniqueName: \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\") pod \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " Oct 30 01:22:16.420935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.420913 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\") pod \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\" (UID: \"19481de2-9d24-474e-abd3-82c1bfa0b2f4\") " Oct 30 01:22:16.422369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.422335 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume" pod "19481de2-9d24-474e-abd3-82c1bfa0b2f4" (UID: "19481de2-9d24-474e-abd3-82c1bfa0b2f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:16.423062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.423043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g" pod "19481de2-9d24-474e-abd3-82c1bfa0b2f4" (UID: "19481de2-9d24-474e-abd3-82c1bfa0b2f4"). InnerVolumeSpecName "kube-api-access-zmh2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:16.523002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.522938 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh2g\" (UniqueName: \"kubernetes.io/projected/19481de2-9d24-474e-abd3-82c1bfa0b2f4-kube-api-access-zmh2g\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:16.523002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.522965 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19481de2-9d24-474e-abd3-82c1bfa0b2f4-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:16.546596 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.546572 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.687966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.687930 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.765840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.765810 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj"] Oct 30 01:22:16.769158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.769136 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-812ba2-zgftj"] Oct 30 01:22:16.866400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:16.866372 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:17.214145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.214073 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" Oct 30 01:22:17.214145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.214096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp" event={"ID":"19481de2-9d24-474e-abd3-82c1bfa0b2f4","Type":"ContainerDied","Data":"5cf3453ba2f1f6e9f73aab4bfd7587e2f26f6f9320689c8d856f9ad160dea83e"} Oct 30 01:22:17.214636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.214153 5938 scope.go:122] "RemoveContainer" containerID="0795254e8effc1c21554934a9bd92f3b6586ba857af5b5f3d72783bf3fde0afe" Oct 30 01:22:17.263360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.263335 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:22:17.274316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.274294 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-812ba2-596b9f7c9d-gtlmp"] Oct 30 01:22:17.348431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.348396 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:17.876276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.876242 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" path="/var/lib/kubelet/pods/19481de2-9d24-474e-abd3-82c1bfa0b2f4/volumes" Oct 30 01:22:17.876687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:17.876673 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1c57f6-762d-4421-b851-be35600fde63" path="/var/lib/kubelet/pods/df1c57f6-762d-4421-b851-be35600fde63/volumes" Oct 30 01:22:18.149502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:18.149416 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:22:18.149744 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:18.149725 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:22:19.026541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.026496 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.138046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138016 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f"] Oct 30 01:22:19.138669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138650 5938 cpu_manager.go:414] "containerMap: removing container" podUID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" containerName="payload-processing" Oct 30 01:22:19.138669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138667 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" containerName="payload-processing" Oct 30 01:22:19.138835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138682 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" Oct 30 01:22:19.138835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138687 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" Oct 30 01:22:19.138835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138800 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="19481de2-9d24-474e-abd3-82c1bfa0b2f4" containerName="payload-processing" Oct 30 01:22:19.138835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.138809 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e777165-f922-436a-a9a5-d21765c8ff20" containerName="main" Oct 30 01:22:19.144041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.144018 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.159757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.159697 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f"] Oct 30 01:22:19.168315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.168292 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:19.168454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.168369 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:22:19.171592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.171570 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.260392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.260358 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.260547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.260399 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7z7c\" (UniqueName: \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.260593 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.260557 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.324657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.324575 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:22:19.324856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.324823 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" containerID="cri-o://fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008" gracePeriod=30 Oct 30 01:22:19.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.362153 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.362328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.362309 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.362411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.362352 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7z7c\" (UniqueName: \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.362989 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.362963 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.364390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.364367 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.385273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.385250 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7z7c\" (UniqueName: \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\") pod \"maas-api-revoke-sub-9f018d58-rsn6f\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.393378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.393345 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.460664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.460225 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:19.536381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.536354 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.776136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:19.776113 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f"] Oct 30 01:22:19.778008 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:19.777984 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cde3538_d4ae_4f83_9a8f_2341b80bf36b.slice/crio-f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100 WatchSource:0}: Error finding container f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100: Status 404 returned error can't find the container with id f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100 Oct 30 01:22:20.241077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:20.241048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" event={"ID":"0cde3538-d4ae-4f83-9a8f-2341b80bf36b","Type":"ContainerStarted","Data":"aaf5372b1bd1ab7412d615037e6d88c1bb862f585f18e382bbedc6f2b98f8d37"} Oct 30 01:22:20.241451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:20.241096 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" event={"ID":"0cde3538-d4ae-4f83-9a8f-2341b80bf36b","Type":"ContainerStarted","Data":"f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100"} Oct 30 01:22:20.281439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:20.281388 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" podStartSLOduration=1.281374305 podStartE2EDuration="1.281374305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:20.278824198 +0000 UTC m=+2992.519059314" watchObservedRunningTime="2026-10-30 01:22:20.281374305 +0000 UTC m=+2992.521609418" Oct 30 01:22:21.249005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:21.248973 5938 generic.go:356] "Generic (PLEG): container finished" podID="0cde3538-d4ae-4f83-9a8f-2341b80bf36b" containerID="aaf5372b1bd1ab7412d615037e6d88c1bb862f585f18e382bbedc6f2b98f8d37" exitCode=0 Oct 30 01:22:21.249390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:21.249055 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" event={"ID":"0cde3538-d4ae-4f83-9a8f-2341b80bf36b","Type":"ContainerDied","Data":"aaf5372b1bd1ab7412d615037e6d88c1bb862f585f18e382bbedc6f2b98f8d37"} Oct 30 01:22:21.574118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:21.574024 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:22.402494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.402470 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:22.493062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.493034 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\") pod \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " Oct 30 01:22:22.493224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.493103 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\" (UniqueName: \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\") pod \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " Oct 30 01:22:22.493224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.493215 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\") pod \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\" (UID: \"0cde3538-d4ae-4f83-9a8f-2341b80bf36b\") " Oct 30 01:22:22.493434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.493413 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca" pod "0cde3538-d4ae-4f83-9a8f-2341b80bf36b" (UID: "0cde3538-d4ae-4f83-9a8f-2341b80bf36b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:22.493714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.493696 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:22.495465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.495437 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c" pod "0cde3538-d4ae-4f83-9a8f-2341b80bf36b" (UID: "0cde3538-d4ae-4f83-9a8f-2341b80bf36b"). InnerVolumeSpecName "kube-api-access-q7z7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:22.495562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.495467 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca" pod "0cde3538-d4ae-4f83-9a8f-2341b80bf36b" (UID: "0cde3538-d4ae-4f83-9a8f-2341b80bf36b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:22.594993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.594943 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7z7c\" (UniqueName: \"kubernetes.io/projected/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-kube-api-access-q7z7c\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:22.594993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.594965 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cde3538-d4ae-4f83-9a8f-2341b80bf36b-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:22.707298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.707271 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2"] Oct 30 01:22:22.707837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.707824 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0cde3538-d4ae-4f83-9a8f-2341b80bf36b" containerName="revoke-keys" Oct 30 01:22:22.707918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.707839 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cde3538-d4ae-4f83-9a8f-2341b80bf36b" containerName="revoke-keys" Oct 30 01:22:22.707965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.707945 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cde3538-d4ae-4f83-9a8f-2341b80bf36b" containerName="revoke-keys" Oct 30 01:22:22.711209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.711193 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.729224 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.729202 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2"] Oct 30 01:22:22.797456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.797431 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.797586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.797460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrdc\" (UniqueName: \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.797646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.797626 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.899115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.899092 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.899227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.899122 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrdc\" (UniqueName: \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.899227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.899214 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.900441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.900404 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.912501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.912479 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:22.921819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:22.921797 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrdc\" (UniqueName: \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\") pod \"maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:23.024459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.024435 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:23.263143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.263078 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" Oct 30 01:22:23.263143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.263087 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f" event={"ID":"0cde3538-d4ae-4f83-9a8f-2341b80bf36b","Type":"ContainerDied","Data":"f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100"} Oct 30 01:22:23.263334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.263161 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f33006e4baeb51c2a5f7807bbcd3c1ebdf2df8847b7bd3ae3a0e260858583100" Oct 30 01:22:23.326285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.326259 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2"] Oct 30 01:22:23.328066 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:23.328040 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89544dfa_7b4f_40e0_867e_b5adebe676b9.slice/crio-315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21 WatchSource:0}: Error finding container 315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21: Status 404 returned error can't find the container with id 315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21 Oct 30 01:22:23.356739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.356715 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:23.504102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.504029 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:23.628178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.628143 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:22:23.628422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.628391 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" containerID="cri-o://9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c" gracePeriod=30 Oct 30 01:22:23.700643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.700609 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:23.702166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.702142 5938 patch_prober.go:29] interesting pod/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:22:23.702268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:23.702188 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:22:24.271329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.271299 5938 generic.go:356] "Generic (PLEG): container finished" podID="89544dfa-7b4f-40e0-867e-b5adebe676b9" containerID="a73a2e9ca2de6e4e3be671860312863567093be93d73cfc84de235bac5c0d1db" exitCode=0 Oct 30 01:22:24.271470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.271386 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" event={"ID":"89544dfa-7b4f-40e0-867e-b5adebe676b9","Type":"ContainerDied","Data":"a73a2e9ca2de6e4e3be671860312863567093be93d73cfc84de235bac5c0d1db"} Oct 30 01:22:24.271470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.271439 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" event={"ID":"89544dfa-7b4f-40e0-867e-b5adebe676b9","Type":"ContainerStarted","Data":"315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21"} Oct 30 01:22:24.599097 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.599074 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:22:24.722914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.722818 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.722914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.722906 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723074 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723175 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723235 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723235 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:24.723308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723279 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723318 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:24.723416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723383 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723486 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723537 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\") pod \"77b0aee2-7805-463c-9c43-472daea37f3c\" (UID: \"77b0aee2-7805-463c-9c43-472daea37f3c\") " Oct 30 01:22:24.723688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723664 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:24.723769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723753 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:24.723933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.723906 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:24.724160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.724136 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.724239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.724165 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.724239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.724181 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.724239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.724196 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b0aee2-7805-463c-9c43-472daea37f3c-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.724239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.724209 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.725290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.725263 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:24.725467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.725443 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:24.725598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.725579 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "kube-api-access-8vfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:24.725666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.725624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token" pod "77b0aee2-7805-463c-9c43-472daea37f3c" (UID: "77b0aee2-7805-463c-9c43-472daea37f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:24.825755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.825722 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b0aee2-7805-463c-9c43-472daea37f3c-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.825755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.825753 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.825755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.825763 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b0aee2-7805-463c-9c43-472daea37f3c-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:24.825965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:24.825774 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vfb8\" (UniqueName: \"kubernetes.io/projected/77b0aee2-7805-463c-9c43-472daea37f3c-kube-api-access-8vfb8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:25.280014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.279981 5938 generic.go:356] "Generic (PLEG): container finished" podID="77b0aee2-7805-463c-9c43-472daea37f3c" containerID="fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008" exitCode=0 Oct 30 01:22:25.280179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.280041 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" Oct 30 01:22:25.280179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.280059 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerDied","Data":"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008"} Oct 30 01:22:25.280179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.280114 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk" event={"ID":"77b0aee2-7805-463c-9c43-472daea37f3c","Type":"ContainerDied","Data":"a43120678ce4bf3c0a67a44602b5f727ac9b9eabffb95bd821c2a16d4449beef"} Oct 30 01:22:25.280179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.280137 5938 scope.go:122] "RemoveContainer" containerID="fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008" Oct 30 01:22:25.293763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.293745 5938 scope.go:122] "RemoveContainer" containerID="221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b" Oct 30 01:22:25.305747 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.305724 5938 scope.go:122] "RemoveContainer" containerID="fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008" Oct 30 01:22:25.306631 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:25.306607 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008\": container with ID starting with fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008 not found: ID does not exist" containerID="fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008" Oct 30 01:22:25.306724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.306642 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008"} err="failed to get container status \"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008\": rpc error: code = NotFound desc = could not find container \"fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008\": container with ID starting with fbe3309e13ec4dd4e763e35e32a032564c96deb0d6bdf270089fe7a2a89d7008 not found: ID does not exist" Oct 30 01:22:25.306724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.306659 5938 scope.go:122] "RemoveContainer" containerID="221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b" Oct 30 01:22:25.307015 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:25.306991 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b\": container with ID starting with 221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b not found: ID does not exist" containerID="221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b" Oct 30 01:22:25.307091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.307019 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b"} err="failed to get container status \"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b\": rpc error: code = NotFound desc = could not find container \"221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b\": container with ID starting with 221a31dda30b429cfa33d3f6d80d9e41603083995af034c0b41622160baf814b not found: ID does not exist" Oct 30 01:22:25.344828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.344783 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:22:25.354661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.354628 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-hmlbk"] Oct 30 01:22:25.422778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.422750 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:25.431341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.431284 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:25.533634 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.533558 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\") pod \"89544dfa-7b4f-40e0-867e-b5adebe676b9\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " Oct 30 01:22:25.533805 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.533786 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\" (UniqueName: \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\") pod \"89544dfa-7b4f-40e0-867e-b5adebe676b9\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " Oct 30 01:22:25.533956 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.533931 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\") pod \"89544dfa-7b4f-40e0-867e-b5adebe676b9\" (UID: \"89544dfa-7b4f-40e0-867e-b5adebe676b9\") " Oct 30 01:22:25.534126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.534004 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca" pod "89544dfa-7b4f-40e0-867e-b5adebe676b9" (UID: "89544dfa-7b4f-40e0-867e-b5adebe676b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:25.534367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.534349 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:25.535902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.535859 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca" pod "89544dfa-7b4f-40e0-867e-b5adebe676b9" (UID: "89544dfa-7b4f-40e0-867e-b5adebe676b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:25.536024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.536001 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc" pod "89544dfa-7b4f-40e0-867e-b5adebe676b9" (UID: "89544dfa-7b4f-40e0-867e-b5adebe676b9"). InnerVolumeSpecName "kube-api-access-tmrdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:25.635552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.635526 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89544dfa-7b4f-40e0-867e-b5adebe676b9-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:25.635552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.635551 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrdc\" (UniqueName: \"kubernetes.io/projected/89544dfa-7b4f-40e0-867e-b5adebe676b9-kube-api-access-tmrdc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:25.876748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:25.876714 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" path="/var/lib/kubelet/pods/77b0aee2-7805-463c-9c43-472daea37f3c/volumes" Oct 30 01:22:26.288978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:26.288900 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" Oct 30 01:22:26.288978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:26.288918 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2" event={"ID":"89544dfa-7b4f-40e0-867e-b5adebe676b9","Type":"ContainerDied","Data":"315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21"} Oct 30 01:22:26.288978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:26.288961 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315d615be8d7f2af605f7649bc74f38627b7c7c08a579121716a491f9ae36c21" Oct 30 01:22:28.073360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:28.073312 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:28.841068 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:28.841026 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:22:28.894121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:28.894062 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:22:29.068585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068556 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.068794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068600 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.068794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068622 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.068794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068713 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.068794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068767 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.069112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068899 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.069112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068932 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.069112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.068945 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.069112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069032 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.069112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069093 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\") pod \"c8899fba-c9b1-4d54-9f33-5708dd58b119\" (UID: \"c8899fba-c9b1-4d54-9f33-5708dd58b119\") " Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069214 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069230 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069288 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069311 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069622 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8899fba-c9b1-4d54-9f33-5708dd58b119-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069645 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069658 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069671 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.069972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.069685 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.071130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.071098 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:29.071246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.071152 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "kube-api-access-dvzgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:29.071246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.071165 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:29.071342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.071328 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy" pod "c8899fba-c9b1-4d54-9f33-5708dd58b119" (UID: "c8899fba-c9b1-4d54-9f33-5708dd58b119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:29.171015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.170960 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.171015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.170987 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzgf\" (UniqueName: \"kubernetes.io/projected/c8899fba-c9b1-4d54-9f33-5708dd58b119-kube-api-access-dvzgf\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.171015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.170996 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.171015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.171005 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8899fba-c9b1-4d54-9f33-5708dd58b119-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:29.311339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.311309 5938 generic.go:356] "Generic (PLEG): container finished" podID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerID="9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c" exitCode=0 Oct 30 01:22:29.311478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.311362 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" Oct 30 01:22:29.311478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.311384 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerDied","Data":"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c"} Oct 30 01:22:29.311478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.311461 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2" event={"ID":"c8899fba-c9b1-4d54-9f33-5708dd58b119","Type":"ContainerDied","Data":"8ab5eb599b93435f2e49c496cc52890c6ef5d83e49489dba4ffee7556a60ef0a"} Oct 30 01:22:29.311622 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.311483 5938 scope.go:122] "RemoveContainer" containerID="9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c" Oct 30 01:22:29.322441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.322423 5938 scope.go:122] "RemoveContainer" containerID="731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228" Oct 30 01:22:29.332054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.332036 5938 scope.go:122] "RemoveContainer" containerID="9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c" Oct 30 01:22:29.332295 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:29.332278 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c\": container with ID starting with 9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c not found: ID does not exist" containerID="9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c" Oct 30 01:22:29.332344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.332306 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c"} err="failed to get container status \"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c\": rpc error: code = NotFound desc = could not find container \"9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c\": container with ID starting with 9cf95962b485100b9b9df6122a281ee2abc56ef69381cda404420ff3d1111a6c not found: ID does not exist" Oct 30 01:22:29.332344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.332322 5938 scope.go:122] "RemoveContainer" containerID="731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228" Oct 30 01:22:29.332537 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:29.332523 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228\": container with ID starting with 731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228 not found: ID does not exist" containerID="731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228" Oct 30 01:22:29.332585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.332541 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228"} err="failed to get container status \"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228\": rpc error: code = NotFound desc = could not find container \"731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228\": container with ID starting with 731cc0ffdf8e2bd0cd8009c316c784a0a9d7028cdbe2a1a0309661280237c228 not found: ID does not exist" Oct 30 01:22:29.362232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.362211 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:22:29.368307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.368289 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-812ba2-openshift-default-779b8b57d7-v75n2"] Oct 30 01:22:29.881604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.881571 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" path="/var/lib/kubelet/pods/c8899fba-c9b1-4d54-9f33-5708dd58b119/volumes" Oct 30 01:22:29.984281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:29.984239 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:30.659974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:30.659936 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:22:30.698737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:30.698699 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:22:30.757817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:30.757782 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:30.902547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:30.902519 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:31.747638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:31.747608 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:33.421725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:33.421693 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:35.712560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:35.712524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.292613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:36.292569 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:36.446004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:36.445968 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.449892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:37.449833 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.587759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:37.587711 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:38.896233 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:38.896167 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:39.470913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:39.470854 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:39.583029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:39.582846 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48"] Oct 30 01:22:39.594011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:39.593988 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8e0f66-r6z48"] Oct 30 01:22:39.617034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:39.617001 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:39.876828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:39.876795 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11896b14-7a0b-40c1-8a5f-188c27b7fc4f" path="/var/lib/kubelet/pods/11896b14-7a0b-40c1-8a5f-188c27b7fc4f/volumes" Oct 30 01:22:40.288124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:40.288040 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.182922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.182719 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:22:41.183145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.182961 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" podUID="870b8015-3078-4cf3-8d88-a77448233bbe" containerName="maas-api" containerID="cri-o://c1ff6e63cfb145648169ef95565c2e045b16cd752ada39c63b9cedc47e325d17" gracePeriod=30 Oct 30 01:22:41.293665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.293631 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.390083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.390049 5938 generic.go:356] "Generic (PLEG): container finished" podID="870b8015-3078-4cf3-8d88-a77448233bbe" containerID="c1ff6e63cfb145648169ef95565c2e045b16cd752ada39c63b9cedc47e325d17" exitCode=0 Oct 30 01:22:41.390254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.390173 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" event={"ID":"870b8015-3078-4cf3-8d88-a77448233bbe","Type":"ContainerDied","Data":"c1ff6e63cfb145648169ef95565c2e045b16cd752ada39c63b9cedc47e325d17"} Oct 30 01:22:41.482298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.482271 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:22:41.503969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.503944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.513271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.513250 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\") pod \"870b8015-3078-4cf3-8d88-a77448233bbe\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " Oct 30 01:22:41.513361 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.513346 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\") pod \"870b8015-3078-4cf3-8d88-a77448233bbe\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " Oct 30 01:22:41.513447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.513431 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\" (UniqueName: \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\") pod \"870b8015-3078-4cf3-8d88-a77448233bbe\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " Oct 30 01:22:41.513507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.513490 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\") pod \"870b8015-3078-4cf3-8d88-a77448233bbe\" (UID: \"870b8015-3078-4cf3-8d88-a77448233bbe\") " Oct 30 01:22:41.513920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.513892 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca" pod "870b8015-3078-4cf3-8d88-a77448233bbe" (UID: "870b8015-3078-4cf3-8d88-a77448233bbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:41.514104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.514079 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870b8015-3078-4cf3-8d88-a77448233bbe-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.515452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.515428 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc" pod "870b8015-3078-4cf3-8d88-a77448233bbe" (UID: "870b8015-3078-4cf3-8d88-a77448233bbe"). InnerVolumeSpecName "kube-api-access-8qtzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:41.515542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.515465 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs" pod "870b8015-3078-4cf3-8d88-a77448233bbe" (UID: "870b8015-3078-4cf3-8d88-a77448233bbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:41.515542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.515502 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls" pod "870b8015-3078-4cf3-8d88-a77448233bbe" (UID: "870b8015-3078-4cf3-8d88-a77448233bbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:41.615614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.615584 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtzc\" (UniqueName: \"kubernetes.io/projected/870b8015-3078-4cf3-8d88-a77448233bbe-kube-api-access-8qtzc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.615614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.615612 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.615614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.615621 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870b8015-3078-4cf3-8d88-a77448233bbe-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:41.701183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.701153 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.777423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.777347 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:22:41.777589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.777549 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" podUID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" containerName="payload-processing" containerID="cri-o://46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f" gracePeriod=30 Oct 30 01:22:41.798009 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.797979 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:22:41.798237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.798209 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" podUID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" containerName="payload-pre-processing" containerID="cri-o://ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0" gracePeriod=30 Oct 30 01:22:41.844649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.844617 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.983505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:41.983467 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.105679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.105472 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:22:42.169640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.169608 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:22:42.180839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.180806 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.227210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.227179 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\" (UniqueName: \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\") pod \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " Oct 30 01:22:42.227366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.227347 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\") pod \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " Oct 30 01:22:42.227446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.227425 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\" (UniqueName: \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\") pod \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\" (UID: \"9d650a5a-9925-485f-9529-2bd5d3fa3de2\") " Oct 30 01:22:42.227584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.227548 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\") pod \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\" (UID: \"94c25193-d31d-48e8-8ef7-ca855ee11ab7\") " Oct 30 01:22:42.228144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.227724 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume" pod "9d650a5a-9925-485f-9529-2bd5d3fa3de2" (UID: "9d650a5a-9925-485f-9529-2bd5d3fa3de2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:42.228144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.228070 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d650a5a-9925-485f-9529-2bd5d3fa3de2-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:42.228144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.228103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume" pod "94c25193-d31d-48e8-8ef7-ca855ee11ab7" (UID: "94c25193-d31d-48e8-8ef7-ca855ee11ab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:42.229388 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.229357 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w" pod "94c25193-d31d-48e8-8ef7-ca855ee11ab7" (UID: "94c25193-d31d-48e8-8ef7-ca855ee11ab7"). InnerVolumeSpecName "kube-api-access-lm95w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:42.230345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.230314 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29" pod "9d650a5a-9925-485f-9529-2bd5d3fa3de2" (UID: "9d650a5a-9925-485f-9529-2bd5d3fa3de2"). InnerVolumeSpecName "kube-api-access-jmc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:42.321717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.321655 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.329154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.329129 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmc29\" (UniqueName: \"kubernetes.io/projected/9d650a5a-9925-485f-9529-2bd5d3fa3de2-kube-api-access-jmc29\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:42.329258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.329156 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c25193-d31d-48e8-8ef7-ca855ee11ab7-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:42.329258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.329172 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm95w\" (UniqueName: \"kubernetes.io/projected/94c25193-d31d-48e8-8ef7-ca855ee11ab7-kube-api-access-lm95w\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:42.399056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.399024 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" Oct 30 01:22:42.399188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.399048 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk" event={"ID":"870b8015-3078-4cf3-8d88-a77448233bbe","Type":"ContainerDied","Data":"f5c5f782e3ef6f0c2eb8e685a87e9c9ef670fc626f5a3e87b6c79916c2b01ee8"} Oct 30 01:22:42.399188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.399119 5938 scope.go:122] "RemoveContainer" containerID="c1ff6e63cfb145648169ef95565c2e045b16cd752ada39c63b9cedc47e325d17" Oct 30 01:22:42.401209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.401181 5938 generic.go:356] "Generic (PLEG): container finished" podID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" containerID="ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0" exitCode=0 Oct 30 01:22:42.401329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.401314 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" Oct 30 01:22:42.402342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.402321 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" event={"ID":"94c25193-d31d-48e8-8ef7-ca855ee11ab7","Type":"ContainerDied","Data":"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0"} Oct 30 01:22:42.402457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.402368 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh" event={"ID":"94c25193-d31d-48e8-8ef7-ca855ee11ab7","Type":"ContainerDied","Data":"08870a66ae523c085361a268a640f148d1f31f6415bbb8ba281448656dc72b70"} Oct 30 01:22:42.403418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.403399 5938 generic.go:356] "Generic (PLEG): container finished" podID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" containerID="46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f" exitCode=0 Oct 30 01:22:42.403500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.403444 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" event={"ID":"9d650a5a-9925-485f-9529-2bd5d3fa3de2","Type":"ContainerDied","Data":"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f"} Oct 30 01:22:42.403500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.403475 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" event={"ID":"9d650a5a-9925-485f-9529-2bd5d3fa3de2","Type":"ContainerDied","Data":"60933842cbce7aa1ac0346a1c08b6a0dc19b30c25b75c4c719a6aaca00f916ec"} Oct 30 01:22:42.403573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.403518 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr" Oct 30 01:22:42.412136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.412111 5938 scope.go:122] "RemoveContainer" containerID="ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0" Oct 30 01:22:42.413318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.413244 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2"] Oct 30 01:22:42.415850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.415827 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b73232-c45t2"] Oct 30 01:22:42.421277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.421262 5938 scope.go:122] "RemoveContainer" containerID="ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0" Oct 30 01:22:42.421518 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:42.421500 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0\": container with ID starting with ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0 not found: ID does not exist" containerID="ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0" Oct 30 01:22:42.421571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.421525 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0"} err="failed to get container status \"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0\": rpc error: code = NotFound desc = could not find container \"ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0\": container with ID starting with ca94460a3f29504de1af6a7ea9cc118542f25e8866acf72071fe3a7331c3d2a0 not found: ID does not exist" Oct 30 01:22:42.421571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.421541 5938 scope.go:122] "RemoveContainer" containerID="46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f" Oct 30 01:22:42.429612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.429595 5938 scope.go:122] "RemoveContainer" containerID="46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f" Oct 30 01:22:42.429899 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:42.429855 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f\": container with ID starting with 46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f not found: ID does not exist" containerID="46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f" Oct 30 01:22:42.429899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.429893 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f"} err="failed to get container status \"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f\": rpc error: code = NotFound desc = could not find container \"46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f\": container with ID starting with 46c96038b216ff90ee85961dcb424aca723bc0295bb60cf4a93003cbc161e80f not found: ID does not exist" Oct 30 01:22:42.462498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.462478 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:22:42.465787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.465765 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.466813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.466794 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-73435a-7556878fbf-cbvlk"] Oct 30 01:22:42.514941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.514916 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:22:42.538908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.538338 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b73232-f54cbdc7b-5mlgr"] Oct 30 01:22:42.565913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.563061 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:22:42.567208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.567173 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b73232-6848d99b47-4xmfh"] Oct 30 01:22:42.854263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.854220 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.990277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.990248 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.997829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.997794 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv"] Oct 30 01:22:42.999904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:42.999884 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6f4fba-vt8fv"] Oct 30 01:22:43.131193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.131119 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.537833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.537758 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.763629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.763591 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.877451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.877419 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b4f97a-ed54-43ba-a81d-faf4dae44f3d" path="/var/lib/kubelet/pods/46b4f97a-ed54-43ba-a81d-faf4dae44f3d/volumes" Oct 30 01:22:43.878024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.878006 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870b8015-3078-4cf3-8d88-a77448233bbe" path="/var/lib/kubelet/pods/870b8015-3078-4cf3-8d88-a77448233bbe/volumes" Oct 30 01:22:43.878557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.878542 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89544dfa-7b4f-40e0-867e-b5adebe676b9" path="/var/lib/kubelet/pods/89544dfa-7b4f-40e0-867e-b5adebe676b9/volumes" Oct 30 01:22:43.879472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.879455 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" path="/var/lib/kubelet/pods/94c25193-d31d-48e8-8ef7-ca855ee11ab7/volumes" Oct 30 01:22:43.880004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.879987 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" path="/var/lib/kubelet/pods/9d650a5a-9925-485f-9529-2bd5d3fa3de2/volumes" Oct 30 01:22:43.906131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:43.906102 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:44.921931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:44.921888 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.584150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:46.584114 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.735600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:46.735559 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.888954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:46.888832 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.890901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:46.890857 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 01:22:46.891087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:46.891065 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" containerID="cri-o://0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726" gracePeriod=30 Oct 30 01:22:47.047633 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.047602 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.477166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.477128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.600008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.599974 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb"] Oct 30 01:22:47.600730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600712 5938 cpu_manager.go:414] "containerMap: removing container" podUID="870b8015-3078-4cf3-8d88-a77448233bbe" containerName="maas-api" Oct 30 01:22:47.600781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600732 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870b8015-3078-4cf3-8d88-a77448233bbe" containerName="maas-api" Oct 30 01:22:47.600781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600750 5938 cpu_manager.go:414] "containerMap: removing container" podUID="89544dfa-7b4f-40e0-867e-b5adebe676b9" containerName="revoke-keys" Oct 30 01:22:47.600781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600758 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89544dfa-7b4f-40e0-867e-b5adebe676b9" containerName="revoke-keys" Oct 30 01:22:47.600781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600771 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:47.600781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600778 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600791 5938 cpu_manager.go:414] "containerMap: removing container" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600798 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600814 5938 cpu_manager.go:414] "containerMap: removing container" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600820 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600835 5938 cpu_manager.go:414] "containerMap: removing container" podUID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" containerName="payload-pre-processing" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600843 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" containerName="payload-pre-processing" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600858 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" containerName="payload-processing" Oct 30 01:22:47.600967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.600880 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" containerName="payload-processing" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601037 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601051 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="870b8015-3078-4cf3-8d88-a77448233bbe" containerName="maas-api" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601061 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="89544dfa-7b4f-40e0-867e-b5adebe676b9" containerName="revoke-keys" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601070 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601080 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b0aee2-7805-463c-9c43-472daea37f3c" containerName="istio-proxy" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601089 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c25193-d31d-48e8-8ef7-ca855ee11ab7" containerName="payload-pre-processing" Oct 30 01:22:47.601214 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.601097 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d650a5a-9925-485f-9529-2bd5d3fa3de2" containerName="payload-processing" Oct 30 01:22:47.607133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.607109 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.621258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.621226 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb"] Oct 30 01:22:47.630151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.630118 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:22:47.630528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.630503 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:22:47.645891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.642376 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:47.645891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.642552 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:22:47.700491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.700461 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.700663 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.700643 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.700719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.700676 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsr79\" (UniqueName: \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.802265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.802190 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.802446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.802303 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.802446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.802328 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsr79\" (UniqueName: \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.802974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.802952 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.804626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.804599 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.827954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.827932 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsr79\" (UniqueName: \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\") pod \"maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:47.921781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:47.921756 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:48.228039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.227936 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb"] Oct 30 01:22:48.231686 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:48.231653 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15893fad_3979_4219_a2df_bbe1bb845cf1.slice/crio-143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296 WatchSource:0}: Error finding container 143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296: Status 404 returned error can't find the container with id 143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296 Oct 30 01:22:48.362365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.362331 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.431050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.431014 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:22:48.431231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.431207 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" podUID="e181b342-30af-4742-b693-64ea1e1721b9" containerName="payload-processing" containerID="cri-o://02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1" gracePeriod=30 Oct 30 01:22:48.446598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.446563 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" event={"ID":"15893fad-3979-4219-a2df-bbe1bb845cf1","Type":"ContainerStarted","Data":"143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296"} Oct 30 01:22:48.450557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.450535 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:22:48.450712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.450693 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" podUID="aac3a356-3f14-49a6-9b15-066b9686107e" containerName="payload-pre-processing" containerID="cri-o://d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f" gracePeriod=30 Oct 30 01:22:48.567264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.567223 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.709366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.709332 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:22:48.710994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.710964 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.753375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.753349 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:22:48.814807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.814774 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\" (UniqueName: \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\") pod \"aac3a356-3f14-49a6-9b15-066b9686107e\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " Oct 30 01:22:48.814960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.814911 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\" (UniqueName: \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\") pod \"e181b342-30af-4742-b693-64ea1e1721b9\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " Oct 30 01:22:48.815043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.815004 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\") pod \"e181b342-30af-4742-b693-64ea1e1721b9\" (UID: \"e181b342-30af-4742-b693-64ea1e1721b9\") " Oct 30 01:22:48.815136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.815118 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\") pod \"aac3a356-3f14-49a6-9b15-066b9686107e\" (UID: \"aac3a356-3f14-49a6-9b15-066b9686107e\") " Oct 30 01:22:48.816039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.816013 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume" pod "aac3a356-3f14-49a6-9b15-066b9686107e" (UID: "aac3a356-3f14-49a6-9b15-066b9686107e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:48.816751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.816726 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume" pod "e181b342-30af-4742-b693-64ea1e1721b9" (UID: "e181b342-30af-4742-b693-64ea1e1721b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:48.818264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.818213 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f" pod "aac3a356-3f14-49a6-9b15-066b9686107e" (UID: "aac3a356-3f14-49a6-9b15-066b9686107e"). InnerVolumeSpecName "kube-api-access-wfx7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:48.819129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.819103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t" pod "e181b342-30af-4742-b693-64ea1e1721b9" (UID: "e181b342-30af-4742-b693-64ea1e1721b9"). InnerVolumeSpecName "kube-api-access-pjh4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:48.917047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.917018 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e181b342-30af-4742-b693-64ea1e1721b9-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:48.917047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.917046 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aac3a356-3f14-49a6-9b15-066b9686107e-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:48.917218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.917061 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfx7f\" (UniqueName: \"kubernetes.io/projected/aac3a356-3f14-49a6-9b15-066b9686107e-kube-api-access-wfx7f\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:48.917218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.917073 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjh4t\" (UniqueName: \"kubernetes.io/projected/e181b342-30af-4742-b693-64ea1e1721b9-kube-api-access-pjh4t\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:48.943317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:48.943242 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:49.084112 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.084044 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:49.455540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.455505 5938 generic.go:356] "Generic (PLEG): container finished" podID="e181b342-30af-4742-b693-64ea1e1721b9" containerID="02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1" exitCode=0 Oct 30 01:22:49.455707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.455552 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" Oct 30 01:22:49.455707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.455582 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" event={"ID":"e181b342-30af-4742-b693-64ea1e1721b9","Type":"ContainerDied","Data":"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1"} Oct 30 01:22:49.455707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.455639 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm" event={"ID":"e181b342-30af-4742-b693-64ea1e1721b9","Type":"ContainerDied","Data":"a4ceea497ecd9d482a4a495473b8f12910cfb113f9eebb6e60c7525482adfc1b"} Oct 30 01:22:49.455707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.455673 5938 scope.go:122] "RemoveContainer" containerID="02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1" Oct 30 01:22:49.457501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.457478 5938 generic.go:356] "Generic (PLEG): container finished" podID="15893fad-3979-4219-a2df-bbe1bb845cf1" containerID="704092c238d9ff6e492394df78342aa28cea992b4a5df2a20efb452e69ec8dd5" exitCode=0 Oct 30 01:22:49.457617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.457560 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" event={"ID":"15893fad-3979-4219-a2df-bbe1bb845cf1","Type":"ContainerDied","Data":"704092c238d9ff6e492394df78342aa28cea992b4a5df2a20efb452e69ec8dd5"} Oct 30 01:22:49.459183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.459158 5938 generic.go:356] "Generic (PLEG): container finished" podID="aac3a356-3f14-49a6-9b15-066b9686107e" containerID="d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f" exitCode=0 Oct 30 01:22:49.459306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.459195 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" Oct 30 01:22:49.459306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.459198 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" event={"ID":"aac3a356-3f14-49a6-9b15-066b9686107e","Type":"ContainerDied","Data":"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f"} Oct 30 01:22:49.459306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.459241 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk" event={"ID":"aac3a356-3f14-49a6-9b15-066b9686107e","Type":"ContainerDied","Data":"7103d5631429567b965546447badf0d65242079b908ffc44c1a3a9c8ab82740f"} Oct 30 01:22:49.467769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.467747 5938 scope.go:122] "RemoveContainer" containerID="02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1" Oct 30 01:22:49.468098 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:49.468079 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1\": container with ID starting with 02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1 not found: ID does not exist" containerID="02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1" Oct 30 01:22:49.468188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.468109 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1"} err="failed to get container status \"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1\": rpc error: code = NotFound desc = could not find container \"02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1\": container with ID starting with 02fdb10d2e0f822181f723cb102403e237cad838dbd149f5b4c355145a05caa1 not found: ID does not exist" Oct 30 01:22:49.468188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.468130 5938 scope.go:122] "RemoveContainer" containerID="d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f" Oct 30 01:22:49.479965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.479948 5938 scope.go:122] "RemoveContainer" containerID="d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f" Oct 30 01:22:49.480241 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:49.480217 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f\": container with ID starting with d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f not found: ID does not exist" containerID="d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f" Oct 30 01:22:49.480289 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.480249 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f"} err="failed to get container status \"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f\": rpc error: code = NotFound desc = could not find container \"d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f\": container with ID starting with d7db1e4dffe60c258c64be16f0d14c347cf1d14c362eed44430b75858cdb5c5f not found: ID does not exist" Oct 30 01:22:49.505993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.505967 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:22:49.518053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.518028 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1e8c0e-649b5b8dff-k47pk"] Oct 30 01:22:49.649724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.649702 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:22:49.670599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.670574 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1e8c0e-6b88cf5b8d-txnvm"] Oct 30 01:22:49.679952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.679920 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:49.876456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.876419 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac3a356-3f14-49a6-9b15-066b9686107e" path="/var/lib/kubelet/pods/aac3a356-3f14-49a6-9b15-066b9686107e/volumes" Oct 30 01:22:49.877027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:49.877004 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e181b342-30af-4742-b693-64ea1e1721b9" path="/var/lib/kubelet/pods/e181b342-30af-4742-b693-64ea1e1721b9/volumes" Oct 30 01:22:50.270786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.270698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.417857 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.417828 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.558007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.557449 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:22:50.558007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.557688 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" containerID="cri-o://1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b" gracePeriod=30 Oct 30 01:22:50.561798 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.561771 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.722012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.721967 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.741047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.741014 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:50.844745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.844643 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\") pod \"15893fad-3979-4219-a2df-bbe1bb845cf1\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " Oct 30 01:22:50.844745 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.844731 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\") pod \"15893fad-3979-4219-a2df-bbe1bb845cf1\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " Oct 30 01:22:50.845057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.844789 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\" (UniqueName: \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\") pod \"15893fad-3979-4219-a2df-bbe1bb845cf1\" (UID: \"15893fad-3979-4219-a2df-bbe1bb845cf1\") " Oct 30 01:22:50.845443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.845414 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca" pod "15893fad-3979-4219-a2df-bbe1bb845cf1" (UID: "15893fad-3979-4219-a2df-bbe1bb845cf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:50.857904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.848042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca" pod "15893fad-3979-4219-a2df-bbe1bb845cf1" (UID: "15893fad-3979-4219-a2df-bbe1bb845cf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:50.861025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.860989 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79" pod "15893fad-3979-4219-a2df-bbe1bb845cf1" (UID: "15893fad-3979-4219-a2df-bbe1bb845cf1"). InnerVolumeSpecName "kube-api-access-rsr79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:50.879629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.879596 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.946621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.946595 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.946621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.946618 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsr79\" (UniqueName: \"kubernetes.io/projected/15893fad-3979-4219-a2df-bbe1bb845cf1-kube-api-access-rsr79\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:50.946777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:50.946631 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15893fad-3979-4219-a2df-bbe1bb845cf1-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:51.141980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.141948 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.300117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300083 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z"] Oct 30 01:22:51.300664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300644 5938 cpu_manager.go:414] "containerMap: removing container" podUID="aac3a356-3f14-49a6-9b15-066b9686107e" containerName="payload-pre-processing" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300666 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac3a356-3f14-49a6-9b15-066b9686107e" containerName="payload-pre-processing" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300688 5938 cpu_manager.go:414] "containerMap: removing container" podUID="15893fad-3979-4219-a2df-bbe1bb845cf1" containerName="revoke-keys" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300697 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15893fad-3979-4219-a2df-bbe1bb845cf1" containerName="revoke-keys" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300708 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300716 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300735 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e181b342-30af-4742-b693-64ea1e1721b9" containerName="payload-processing" Oct 30 01:22:51.300782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300740 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e181b342-30af-4742-b693-64ea1e1721b9" containerName="payload-processing" Oct 30 01:22:51.301164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300836 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e181b342-30af-4742-b693-64ea1e1721b9" containerName="payload-processing" Oct 30 01:22:51.301164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300851 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8899fba-c9b1-4d54-9f33-5708dd58b119" containerName="istio-proxy" Oct 30 01:22:51.301164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300874 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac3a356-3f14-49a6-9b15-066b9686107e" containerName="payload-pre-processing" Oct 30 01:22:51.301164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.300884 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="15893fad-3979-4219-a2df-bbe1bb845cf1" containerName="revoke-keys" Oct 30 01:22:51.304300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.304279 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.305076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.305052 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.326038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.326016 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z"] Oct 30 01:22:51.451680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.451593 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.451891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.451720 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.451992 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.451966 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwtt\" (UniqueName: \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.475046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.475018 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" Oct 30 01:22:51.475195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.475061 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb" event={"ID":"15893fad-3979-4219-a2df-bbe1bb845cf1","Type":"ContainerDied","Data":"143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296"} Oct 30 01:22:51.475195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.475110 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143da80a941aee2c6fe78d9277f90cecbc6203bb8f5b37d789df5a7566e86296" Oct 30 01:22:51.522357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.522321 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.554995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.553046 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwtt\" (UniqueName: \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.554995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.553172 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.554995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.553239 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.554995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.554132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.558031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.557979 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.577368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.577339 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwtt\" (UniqueName: \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.617514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.617484 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:51.917126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:51.917101 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z"] Oct 30 01:22:51.919603 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:51.919577 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4396ac85_318b_49f3_bd86_7d9ed463d495.slice/crio-27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c WatchSource:0}: Error finding container 27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c: Status 404 returned error can't find the container with id 27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c Oct 30 01:22:52.204630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.204608 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 01:22:52.361815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.361785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.361973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.361825 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.361973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.361917 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.361973 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.361937 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362004 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362078 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362147 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362162 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:52.362184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\") pod \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\" (UID: \"b9453dd2-521d-4380-ba5d-b2208bd3dccf\") " Oct 30 01:22:52.362563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362418 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:52.362563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362433 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:52.362563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362514 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:52.362700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362672 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.362700 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362698 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.362818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362712 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.362818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362726 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.362818 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.362752 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.364538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.364502 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:52.364649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.364553 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:52.364649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.364582 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "kube-api-access-4sbq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:52.364649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.364603 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token" pod "b9453dd2-521d-4380-ba5d-b2208bd3dccf" (UID: "b9453dd2-521d-4380-ba5d-b2208bd3dccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:52.368211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.368187 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 01:22:52.382501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.382467 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 01:22:52.463648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.463595 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbq8\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-kube-api-access-4sbq8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.463648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.463618 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.463648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.463627 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.463648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.463635 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9453dd2-521d-4380-ba5d-b2208bd3dccf-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:52.482642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.482614 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" event={"ID":"4396ac85-318b-49f3-bd86-7d9ed463d495","Type":"ContainerStarted","Data":"3dfa4b082b6e0e72d1d5775bd8da4d293871a6fb57cfb5a2050fe1dd056b1a32"} Oct 30 01:22:52.482741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.482657 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" event={"ID":"4396ac85-318b-49f3-bd86-7d9ed463d495","Type":"ContainerStarted","Data":"27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c"} Oct 30 01:22:52.484572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.484549 5938 generic.go:356] "Generic (PLEG): container finished" podID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerID="0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726" exitCode=0 Oct 30 01:22:52.484669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.484585 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" Oct 30 01:22:52.484669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.484627 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerDied","Data":"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726"} Oct 30 01:22:52.484797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.484671 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q" event={"ID":"b9453dd2-521d-4380-ba5d-b2208bd3dccf","Type":"ContainerDied","Data":"99c0bdf7968f853d67cea6eb37855f860d636f3818753399bb40d52bcc3ae125"} Oct 30 01:22:52.484797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.484698 5938 scope.go:122] "RemoveContainer" containerID="0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726" Oct 30 01:22:52.495028 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.494940 5938 scope.go:122] "RemoveContainer" containerID="1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d" Oct 30 01:22:52.504533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.504424 5938 scope.go:122] "RemoveContainer" containerID="0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726" Oct 30 01:22:52.505758 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:52.505731 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726\": container with ID starting with 0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726 not found: ID does not exist" containerID="0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726" Oct 30 01:22:52.505858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.505768 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726"} err="failed to get container status \"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726\": rpc error: code = NotFound desc = could not find container \"0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726\": container with ID starting with 0c883ad8be1c7d2bbb24dd4814a962b399cd498c92f6c3819cc22ebc35c6e726 not found: ID does not exist" Oct 30 01:22:52.505858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.505791 5938 scope.go:122] "RemoveContainer" containerID="1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d" Oct 30 01:22:52.506134 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:52.506109 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d\": container with ID starting with 1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d not found: ID does not exist" containerID="1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d" Oct 30 01:22:52.506298 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.506147 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d"} err="failed to get container status \"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d\": rpc error: code = NotFound desc = could not find container \"1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d\": container with ID starting with 1ae0bd541538d65051bf05028f176977421b571cece649de0e6ac42463833a8d not found: ID does not exist" Oct 30 01:22:52.522856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.522812 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" podStartSLOduration=1.522801624 podStartE2EDuration="1.522801624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:52.52123503 +0000 UTC m=+3024.761470176" watchObservedRunningTime="2026-10-30 01:22:52.522801624 +0000 UTC m=+3024.763036737" Oct 30 01:22:52.532531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.532512 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:52.556278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.556256 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 01:22:52.559942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:52.559922 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8e0f66-openshift-default-75fdc5ccf7-c7q4q"] Oct 30 01:22:53.228525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:53.228441 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.493444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:53.493367 5938 generic.go:356] "Generic (PLEG): container finished" podID="4396ac85-318b-49f3-bd86-7d9ed463d495" containerID="3dfa4b082b6e0e72d1d5775bd8da4d293871a6fb57cfb5a2050fe1dd056b1a32" exitCode=0 Oct 30 01:22:53.493615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:53.493450 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" event={"ID":"4396ac85-318b-49f3-bd86-7d9ed463d495","Type":"ContainerDied","Data":"3dfa4b082b6e0e72d1d5775bd8da4d293871a6fb57cfb5a2050fe1dd056b1a32"} Oct 30 01:22:53.876706 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:53.876680 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" path="/var/lib/kubelet/pods/b9453dd2-521d-4380-ba5d-b2208bd3dccf/volumes" Oct 30 01:22:54.652316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.652287 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:54.790625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.790550 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\") pod \"4396ac85-318b-49f3-bd86-7d9ed463d495\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " Oct 30 01:22:54.790788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.790652 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\" (UniqueName: \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\") pod \"4396ac85-318b-49f3-bd86-7d9ed463d495\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " Oct 30 01:22:54.790788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.790759 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\") pod \"4396ac85-318b-49f3-bd86-7d9ed463d495\" (UID: \"4396ac85-318b-49f3-bd86-7d9ed463d495\") " Oct 30 01:22:54.791142 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.791118 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca" pod "4396ac85-318b-49f3-bd86-7d9ed463d495" (UID: "4396ac85-318b-49f3-bd86-7d9ed463d495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:54.791284 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.791269 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:54.792672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.792648 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca" pod "4396ac85-318b-49f3-bd86-7d9ed463d495" (UID: "4396ac85-318b-49f3-bd86-7d9ed463d495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:54.792763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.792740 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt" pod "4396ac85-318b-49f3-bd86-7d9ed463d495" (UID: "4396ac85-318b-49f3-bd86-7d9ed463d495"). InnerVolumeSpecName "kube-api-access-qxwtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:54.891986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.891959 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwtt\" (UniqueName: \"kubernetes.io/projected/4396ac85-318b-49f3-bd86-7d9ed463d495-kube-api-access-qxwtt\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:54.891986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.891985 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4396ac85-318b-49f3-bd86-7d9ed463d495-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:54.957602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:54.957565 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:55.508608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:55.508576 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" Oct 30 01:22:55.508786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:55.508609 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z" event={"ID":"4396ac85-318b-49f3-bd86-7d9ed463d495","Type":"ContainerDied","Data":"27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c"} Oct 30 01:22:55.508786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:55.508656 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d9f3d724b4727f4f3c667ebbec5b65915707ea397a816e86e2ea98f181538c" Oct 30 01:22:55.816907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:55.816883 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:22:55.883711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:55.883595 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.003848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.003818 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.003888 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.003940 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.003976 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004019 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004078 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004143 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004198 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004226 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004263 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004316 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\") pod \"41579ccd-347f-402a-9fa5-19dc639875e9\" (UID: \"41579ccd-347f-402a-9fa5-19dc639875e9\") " Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004560 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:56.004646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004570 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.004763 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.005038 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.005062 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41579ccd-347f-402a-9fa5-19dc639875e9-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.005076 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.005090 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.005179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.005104 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.006651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.006624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "kube-api-access-vtlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:56.006975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.006950 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:22:56.007123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.007103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:56.007186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.007108 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token" pod "41579ccd-347f-402a-9fa5-19dc639875e9" (UID: "41579ccd-347f-402a-9fa5-19dc639875e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:56.027531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.027506 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.099636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.099577 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 01:22:56.105216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.105193 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 01:22:56.105835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.105775 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.105835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.105800 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41579ccd-347f-402a-9fa5-19dc639875e9-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.105835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.105812 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlvl\" (UniqueName: \"kubernetes.io/projected/41579ccd-347f-402a-9fa5-19dc639875e9-kube-api-access-vtlvl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.105835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.105825 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41579ccd-347f-402a-9fa5-19dc639875e9-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:56.114515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.114493 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 01:22:56.244191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.244160 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.285771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.285745 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn"] Oct 30 01:22:56.286448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286430 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4396ac85-318b-49f3-bd86-7d9ed463d495" containerName="revoke-keys" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286450 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4396ac85-318b-49f3-bd86-7d9ed463d495" containerName="revoke-keys" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286486 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286493 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286503 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286508 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286517 5938 cpu_manager.go:414] "containerMap: removing container" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" Oct 30 01:22:56.286536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286522 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" Oct 30 01:22:56.286836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286604 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.286836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286611 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" containerName="istio-proxy" Oct 30 01:22:56.286836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.286621 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4396ac85-318b-49f3-bd86-7d9ed463d495" containerName="revoke-keys" Oct 30 01:22:56.290070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.290047 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.306657 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.306633 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:22:56.307797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.307776 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:22:56.307947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.307811 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:22:56.307947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.307854 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:22:56.323903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.312111 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn"] Oct 30 01:22:56.384855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.384825 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.408912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.408878 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.409050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.408925 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbkd\" (UniqueName: \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.409050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.409003 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.510219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.510193 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.510343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.510238 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbkd\" (UniqueName: \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.510343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.510279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.510946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.510927 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.512873 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.512839 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.518098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.518073 5938 generic.go:356] "Generic (PLEG): container finished" podID="41579ccd-347f-402a-9fa5-19dc639875e9" containerID="1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b" exitCode=0 Oct 30 01:22:56.518180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.518128 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" Oct 30 01:22:56.518180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.518150 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" event={"ID":"41579ccd-347f-402a-9fa5-19dc639875e9","Type":"ContainerDied","Data":"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b"} Oct 30 01:22:56.518285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.518205 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb" event={"ID":"41579ccd-347f-402a-9fa5-19dc639875e9","Type":"ContainerDied","Data":"e16de1699710712e6e5f15a10d810c1d68ea53039378b1e7f8104da2ae892d17"} Oct 30 01:22:56.518285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.518228 5938 scope.go:122] "RemoveContainer" containerID="1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b" Oct 30 01:22:56.529905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.529885 5938 scope.go:122] "RemoveContainer" containerID="1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b" Oct 30 01:22:56.530157 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:56.530137 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b\": container with ID starting with 1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b not found: ID does not exist" containerID="1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b" Oct 30 01:22:56.530218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.530166 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b"} err="failed to get container status \"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b\": rpc error: code = NotFound desc = could not find container \"1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b\": container with ID starting with 1d87a4ede1d98725f1f71228861545955a8a931ce71471b8ee56b139d2a4823b not found: ID does not exist" Oct 30 01:22:56.533193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.533168 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbkd\" (UniqueName: \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\") pod \"maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.568619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.568594 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:22:56.572360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.572339 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-2mhcb"] Oct 30 01:22:56.604258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.604237 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:56.778922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.778888 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng"] Oct 30 01:22:56.779753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.779597 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9453dd2-521d-4380-ba5d-b2208bd3dccf" containerName="istio-proxy" Oct 30 01:22:56.784993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.784974 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:56.802637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.802611 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng"] Oct 30 01:22:56.904023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.903820 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn"] Oct 30 01:22:56.906813 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:56.906779 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48b8b7c_cd56_48a7_8285_c28ff7a4bfd7.slice/crio-12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941 WatchSource:0}: Error finding container 12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941: Status 404 returned error can't find the container with id 12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941 Oct 30 01:22:56.916296 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.916271 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhml\" (UniqueName: \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:56.916378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.916302 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:56.916378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:56.916346 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.017272 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.017249 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhml\" (UniqueName: \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.017387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.017279 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.017387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.017313 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.017917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.017897 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.019504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.019480 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.039995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.039969 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhml\" (UniqueName: \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.113813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.113790 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:57.386149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.386118 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:57.413170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.413142 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng"] Oct 30 01:22:57.415273 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:22:57.415246 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270a2d2a_8be6_46e8_9329_7052e823986c.slice/crio-8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba WatchSource:0}: Error finding container 8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba: Status 404 returned error can't find the container with id 8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba Oct 30 01:22:57.526698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.526669 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" event={"ID":"270a2d2a-8be6-46e8-9329-7052e823986c","Type":"ContainerStarted","Data":"8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba"} Oct 30 01:22:57.528204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.528177 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" event={"ID":"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7","Type":"ContainerStarted","Data":"b5b2d8108c0925aca3cee342775e676460b2b0d91d67d1439745c2fa3a4e8972"} Oct 30 01:22:57.528294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.528220 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" event={"ID":"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7","Type":"ContainerStarted","Data":"12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941"} Oct 30 01:22:57.567476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.567342 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" podStartSLOduration=1.5673288269999999 podStartE2EDuration="1.567328827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:57.567307766 +0000 UTC m=+3029.807542880" watchObservedRunningTime="2026-10-30 01:22:57.567328827 +0000 UTC m=+3029.807563939" Oct 30 01:22:57.875572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:57.875546 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41579ccd-347f-402a-9fa5-19dc639875e9" path="/var/lib/kubelet/pods/41579ccd-347f-402a-9fa5-19dc639875e9/volumes" Oct 30 01:22:58.179850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.179819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:58.318678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.318648 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:58.535239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.535169 5938 generic.go:356] "Generic (PLEG): container finished" podID="270a2d2a-8be6-46e8-9329-7052e823986c" containerID="8f11fdb4052cdc27cfa01c566d16d4c3fadd2788433fb5f773dc3c1888fe9420" exitCode=0 Oct 30 01:22:58.535404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.535255 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" event={"ID":"270a2d2a-8be6-46e8-9329-7052e823986c","Type":"ContainerDied","Data":"8f11fdb4052cdc27cfa01c566d16d4c3fadd2788433fb5f773dc3c1888fe9420"} Oct 30 01:22:58.536852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.536824 5938 generic.go:356] "Generic (PLEG): container finished" podID="a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" containerID="b5b2d8108c0925aca3cee342775e676460b2b0d91d67d1439745c2fa3a4e8972" exitCode=0 Oct 30 01:22:58.537001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:58.536882 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" event={"ID":"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7","Type":"ContainerDied","Data":"b5b2d8108c0925aca3cee342775e676460b2b0d91d67d1439745c2fa3a4e8972"} Oct 30 01:22:58.835092 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:22:58.835003 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:22:59.075584 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.075554 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.509027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.508982 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.688492 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.688466 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:22:59.723285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.723264 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:22:59.848627 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848559 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\" (UniqueName: \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\") pod \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " Oct 30 01:22:59.848770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848627 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\") pod \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " Oct 30 01:22:59.848770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848663 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\") pod \"270a2d2a-8be6-46e8-9329-7052e823986c\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " Oct 30 01:22:59.848770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848685 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\" (UniqueName: \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\") pod \"270a2d2a-8be6-46e8-9329-7052e823986c\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " Oct 30 01:22:59.848947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848923 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\") pod \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\" (UID: \"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7\") " Oct 30 01:22:59.849087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.848982 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\") pod \"270a2d2a-8be6-46e8-9329-7052e823986c\" (UID: \"270a2d2a-8be6-46e8-9329-7052e823986c\") " Oct 30 01:22:59.849087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.849051 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca" pod "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" (UID: "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:59.849087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.849080 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca" pod "270a2d2a-8be6-46e8-9329-7052e823986c" (UID: "270a2d2a-8be6-46e8-9329-7052e823986c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:59.849478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.849451 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:59.849599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.849480 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:59.851042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.851015 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd" pod "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" (UID: "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7"). InnerVolumeSpecName "kube-api-access-2rbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:59.851132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.851014 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca" pod "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" (UID: "a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:59.851132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.851062 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml" pod "270a2d2a-8be6-46e8-9329-7052e823986c" (UID: "270a2d2a-8be6-46e8-9329-7052e823986c"). InnerVolumeSpecName "kube-api-access-nnhml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:59.851132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.851071 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca" pod "270a2d2a-8be6-46e8-9329-7052e823986c" (UID: "270a2d2a-8be6-46e8-9329-7052e823986c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:22:59.950241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.950221 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:59.950241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.950241 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270a2d2a-8be6-46e8-9329-7052e823986c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:59.950367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.950251 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbkd\" (UniqueName: \"kubernetes.io/projected/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7-kube-api-access-2rbkd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:22:59.950367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:22:59.950261 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnhml\" (UniqueName: \"kubernetes.io/projected/270a2d2a-8be6-46e8-9329-7052e823986c-kube-api-access-nnhml\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:00.550117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.550082 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" Oct 30 01:23:00.550510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.550112 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng" event={"ID":"270a2d2a-8be6-46e8-9329-7052e823986c","Type":"ContainerDied","Data":"8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba"} Oct 30 01:23:00.550510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.550159 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ced0b88cdfdbe6b288c354994b94b162702204da2f7cbf5381c7fdd326c12ba" Oct 30 01:23:00.551720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.551697 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" Oct 30 01:23:00.551720 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.551701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn" event={"ID":"a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7","Type":"ContainerDied","Data":"12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941"} Oct 30 01:23:00.551954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.551762 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12b44bdc0081e311ac6c7e944e9cb3c501906866299f0f80513ee2159019b941" Oct 30 01:23:00.910882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.910846 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b"] Oct 30 01:23:00.911358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911334 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" containerName="revoke-keys" Oct 30 01:23:00.911358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911355 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" containerName="revoke-keys" Oct 30 01:23:00.911358 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911365 5938 cpu_manager.go:414] "containerMap: removing container" podUID="270a2d2a-8be6-46e8-9329-7052e823986c" containerName="revoke-keys" Oct 30 01:23:00.911553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911370 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270a2d2a-8be6-46e8-9329-7052e823986c" containerName="revoke-keys" Oct 30 01:23:00.911553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911503 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" containerName="revoke-keys" Oct 30 01:23:00.911553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.911510 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="270a2d2a-8be6-46e8-9329-7052e823986c" containerName="revoke-keys" Oct 30 01:23:00.914782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.914763 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:00.932098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.932079 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b"] Oct 30 01:23:00.936628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.936603 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp"] Oct 30 01:23:00.940271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.940248 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:00.941367 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.941345 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:00.941474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.941417 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:00.941534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.941450 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:00.944418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.944397 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:00.958507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.958484 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp"] Oct 30 01:23:00.962807 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.962786 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:00.962914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.962841 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:00.963007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.962992 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsm4\" (UniqueName: \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:00.963084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.963024 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spct\" (UniqueName: \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:00.963203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.963175 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:00.963270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:00.963238 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.065338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065306 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.065450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065389 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.065450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsm4\" (UniqueName: \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.065578 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065484 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spct\" (UniqueName: \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.065632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065594 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.065632 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.065625 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.066513 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.066481 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.069573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.069543 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.072266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.072239 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.073499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.073468 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.095098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.095069 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsm4\" (UniqueName: \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\") pod \"maas-api-revoke-sub-918f7a86-b5ckp\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.095185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.095103 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spct\" (UniqueName: \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\") pod \"maas-api-revoke-sub-3e2a0a5a-xpz5b\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.227849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.227798 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:01.254451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.254429 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:01.532894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.532850 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b"] Oct 30 01:23:01.534966 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:01.534933 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2ebccf_3746_4731_9935_efbc9f4ab00c.slice/crio-045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a WatchSource:0}: Error finding container 045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a: Status 404 returned error can't find the container with id 045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a Oct 30 01:23:01.540572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.540547 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:01.565811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.565638 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" event={"ID":"3f2ebccf-3746-4731-9935-efbc9f4ab00c","Type":"ContainerStarted","Data":"045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a"} Oct 30 01:23:01.567439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.567416 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp"] Oct 30 01:23:01.572333 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:01.572289 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ea663f_75a1_4abb_bb7b_c94702adf2d4.slice/crio-9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461 WatchSource:0}: Error finding container 9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461: Status 404 returned error can't find the container with id 9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461 Oct 30 01:23:01.947953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:01.947923 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:02.095098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.095002 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:02.574163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.574135 5938 generic.go:356] "Generic (PLEG): container finished" podID="69ea663f-75a1-4abb-bb7b-c94702adf2d4" containerID="a1808fa8be50a357fa6b84feb5115569bdb259c79cfc65ca6975e6356d649ff7" exitCode=0 Oct 30 01:23:02.574547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.574215 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" event={"ID":"69ea663f-75a1-4abb-bb7b-c94702adf2d4","Type":"ContainerDied","Data":"a1808fa8be50a357fa6b84feb5115569bdb259c79cfc65ca6975e6356d649ff7"} Oct 30 01:23:02.574547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.574264 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" event={"ID":"69ea663f-75a1-4abb-bb7b-c94702adf2d4","Type":"ContainerStarted","Data":"9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461"} Oct 30 01:23:02.575965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.575942 5938 generic.go:356] "Generic (PLEG): container finished" podID="3f2ebccf-3746-4731-9935-efbc9f4ab00c" containerID="2a46e706d5d36c6d704f2b2875aa5f31f0b90f402809449b5b1d320d9223801f" exitCode=0 Oct 30 01:23:02.576055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:02.576021 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" event={"ID":"3f2ebccf-3746-4731-9935-efbc9f4ab00c","Type":"ContainerDied","Data":"2a46e706d5d36c6d704f2b2875aa5f31f0b90f402809449b5b1d320d9223801f"} Oct 30 01:23:03.588555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.588525 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:03.736407 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.736381 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:03.765769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.765745 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:03.792736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.792701 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\" (UniqueName: \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\") pod \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " Oct 30 01:23:03.792876 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.792755 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\") pod \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " Oct 30 01:23:03.792876 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.792810 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\") pod \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " Oct 30 01:23:03.792876 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.792857 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\") pod \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\" (UID: \"69ea663f-75a1-4abb-bb7b-c94702adf2d4\") " Oct 30 01:23:03.793100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.792973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\" (UniqueName: \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\") pod \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " Oct 30 01:23:03.793100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.793011 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\") pod \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\" (UID: \"3f2ebccf-3746-4731-9935-efbc9f4ab00c\") " Oct 30 01:23:03.793100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.793078 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca" pod "69ea663f-75a1-4abb-bb7b-c94702adf2d4" (UID: "69ea663f-75a1-4abb-bb7b-c94702adf2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:03.793531 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.793505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca" pod "3f2ebccf-3746-4731-9935-efbc9f4ab00c" (UID: "3f2ebccf-3746-4731-9935-efbc9f4ab00c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:03.793688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.793663 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:03.793833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.793817 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:03.795336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.795302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4" pod "69ea663f-75a1-4abb-bb7b-c94702adf2d4" (UID: "69ea663f-75a1-4abb-bb7b-c94702adf2d4"). InnerVolumeSpecName "kube-api-access-dwsm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:03.795432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.795399 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca" pod "69ea663f-75a1-4abb-bb7b-c94702adf2d4" (UID: "69ea663f-75a1-4abb-bb7b-c94702adf2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:03.795432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.795414 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca" pod "3f2ebccf-3746-4731-9935-efbc9f4ab00c" (UID: "3f2ebccf-3746-4731-9935-efbc9f4ab00c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:03.796081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.796058 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct" pod "3f2ebccf-3746-4731-9935-efbc9f4ab00c" (UID: "3f2ebccf-3746-4731-9935-efbc9f4ab00c"). InnerVolumeSpecName "kube-api-access-7spct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:03.895362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.895339 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsm4\" (UniqueName: \"kubernetes.io/projected/69ea663f-75a1-4abb-bb7b-c94702adf2d4-kube-api-access-dwsm4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:03.895362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.895363 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f2ebccf-3746-4731-9935-efbc9f4ab00c-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:03.895480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.895373 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea663f-75a1-4abb-bb7b-c94702adf2d4-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:03.895480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:03.895381 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spct\" (UniqueName: \"kubernetes.io/projected/3f2ebccf-3746-4731-9935-efbc9f4ab00c-kube-api-access-7spct\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:04.220376 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.220316 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:23:04.220538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.220509 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" containerID="cri-o://cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e" gracePeriod=30 Oct 30 01:23:04.231699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.231674 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:23:04.232067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.232040 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" containerID="cri-o://66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6" gracePeriod=30 Oct 30 01:23:04.590625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.590540 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" Oct 30 01:23:04.590625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.590546 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp" event={"ID":"69ea663f-75a1-4abb-bb7b-c94702adf2d4","Type":"ContainerDied","Data":"9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461"} Oct 30 01:23:04.590625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.590622 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e10cc4b1fbb5ef503bb609c947c582bdad2492391fb5f8b6509d34438792461" Oct 30 01:23:04.592101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.592076 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" event={"ID":"3f2ebccf-3746-4731-9935-efbc9f4ab00c","Type":"ContainerDied","Data":"045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a"} Oct 30 01:23:04.592229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.592117 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045b189f3c191ad773aaea229d912766b8a7d82c47439686b53742050075303a" Oct 30 01:23:04.592229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:04.592086 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b" Oct 30 01:23:05.397140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:05.397108 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.046543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.046509 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.190551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.190518 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:23:07.371901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.371850 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.389011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.388978 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" containerID="cri-o://7be5ac414b094dad1b4d200282fcc225889754eaf6653cc88d8c9186b92e9468" gracePeriod=45 Oct 30 01:23:07.402751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.402704 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" containerID="cri-o://3e63f5cae5dd687d721bb29ab119a768d91282fbf7adfffaaa1d501a06d028fa" gracePeriod=45 Oct 30 01:23:07.518292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.518252 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.616755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.616724 5938 generic.go:356] "Generic (PLEG): container finished" podID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerID="3e63f5cae5dd687d721bb29ab119a768d91282fbf7adfffaaa1d501a06d028fa" exitCode=0 Oct 30 01:23:07.616935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.616911 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" event={"ID":"5160a8f7-8d30-4938-81b6-7140cd82c4db","Type":"ContainerDied","Data":"3e63f5cae5dd687d721bb29ab119a768d91282fbf7adfffaaa1d501a06d028fa"} Oct 30 01:23:07.622549 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.622494 5938 generic.go:356] "Generic (PLEG): container finished" podID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerID="7be5ac414b094dad1b4d200282fcc225889754eaf6653cc88d8c9186b92e9468" exitCode=0 Oct 30 01:23:07.622673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.622583 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" event={"ID":"2160acac-590a-4dd3-bb22-23e6f85ffcb7","Type":"ContainerDied","Data":"7be5ac414b094dad1b4d200282fcc225889754eaf6653cc88d8c9186b92e9468"} Oct 30 01:23:07.681760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.681735 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 01:23:07.699730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.699710 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 01:23:07.748118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748078 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\" (UniqueName: \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748169 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748277 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748259 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.748429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748306 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748338 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\" (UniqueName: \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.748601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748571 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.748693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748673 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.748764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748720 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\" (UniqueName: \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.748764 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748749 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748923 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748790 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748923 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\") pod \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\" (UID: \"2160acac-590a-4dd3-bb22-23e6f85ffcb7\") " Oct 30 01:23:07.748923 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.748900 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\") pod \"5160a8f7-8d30-4938-81b6-7140cd82c4db\" (UID: \"5160a8f7-8d30-4938-81b6-7140cd82c4db\") " Oct 30 01:23:07.749550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749324 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.749651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749571 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.749731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749709 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.749803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749738 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.749803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749762 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.750198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.749979 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.750198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.750164 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.751614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.751557 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.752412 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.752386 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "kube-api-access-jzfd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:07.753617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.753590 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:07.753756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.753631 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.753828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.753754 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm" pod "2160acac-590a-4dd3-bb22-23e6f85ffcb7" (UID: "2160acac-590a-4dd3-bb22-23e6f85ffcb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:07.754318 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.754294 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "kube-api-access-f8lcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:07.754417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.754316 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs" pod "5160a8f7-8d30-4938-81b6-7140cd82c4db" (UID: "5160a8f7-8d30-4938-81b6-7140cd82c4db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:07.850858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850830 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.850858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850856 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2160acac-590a-4dd3-bb22-23e6f85ffcb7-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850886 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lcp\" (UniqueName: \"kubernetes.io/projected/5160a8f7-8d30-4938-81b6-7140cd82c4db-kube-api-access-f8lcp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850897 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850906 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5160a8f7-8d30-4938-81b6-7140cd82c4db-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850914 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5160a8f7-8d30-4938-81b6-7140cd82c4db-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850922 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850929 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850936 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2160acac-590a-4dd3-bb22-23e6f85ffcb7-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:07.851064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:07.850944 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzfd8\" (UniqueName: \"kubernetes.io/projected/2160acac-590a-4dd3-bb22-23e6f85ffcb7-kube-api-access-jzfd8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:08.631067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.631034 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" Oct 30 01:23:08.631527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.631065 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz" event={"ID":"5160a8f7-8d30-4938-81b6-7140cd82c4db","Type":"ContainerDied","Data":"e40b000c4928386b9b91e13c44c28e105ce0699cee67e2025f7a0fbc0848d794"} Oct 30 01:23:08.631527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.631108 5938 scope.go:122] "RemoveContainer" containerID="3e63f5cae5dd687d721bb29ab119a768d91282fbf7adfffaaa1d501a06d028fa" Oct 30 01:23:08.632976 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.632949 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" event={"ID":"2160acac-590a-4dd3-bb22-23e6f85ffcb7","Type":"ContainerDied","Data":"e8291e64593a0ddc9d830bf16a4ea8f0f14318b8d125b740839f995696e5e49d"} Oct 30 01:23:08.633134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.633043 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv" Oct 30 01:23:08.640783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.640762 5938 scope.go:122] "RemoveContainer" containerID="7be5ac414b094dad1b4d200282fcc225889754eaf6653cc88d8c9186b92e9468" Oct 30 01:23:08.680557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.680507 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 01:23:08.692077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.692050 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-2-w3-cefb27-kserve-678db6d6bb-v2qtv"] Oct 30 01:23:08.716607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.716583 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 01:23:08.720756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.720737 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/premium-w3-cefb27-kserve-5c5fd96f8-8c7qz"] Oct 30 01:23:08.930698 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:08.930626 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.491521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.491467 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:23:09.535459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.535437 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:23:09.569409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569377 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569449 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569497 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569533 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569589 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569643 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.569830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569661 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569728 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.569830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569770 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.570066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.569850 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\") pod \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\" (UID: \"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b\") " Oct 30 01:23:09.570066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570034 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:09.570794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570379 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.570794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570404 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.570794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570543 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.570794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570618 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.570794 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.570748 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.572423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.572398 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:09.572637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.572613 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.573038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.573019 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "kube-api-access-z48wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.573300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.573280 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token" pod "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" (UID: "c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.641824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.641792 5938 generic.go:356] "Generic (PLEG): container finished" podID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerID="cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e" exitCode=0 Oct 30 01:23:09.642249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.641839 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" Oct 30 01:23:09.642249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.641906 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" event={"ID":"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b","Type":"ContainerDied","Data":"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e"} Oct 30 01:23:09.642249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.641956 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4" event={"ID":"c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b","Type":"ContainerDied","Data":"8c8594a51644458df2d543a511903b8c73cacfbf90ec7ad74bb2cb2e9b5c1c5c"} Oct 30 01:23:09.642249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.641985 5938 scope.go:122] "RemoveContainer" containerID="cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e" Oct 30 01:23:09.645164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.645140 5938 generic.go:356] "Generic (PLEG): container finished" podID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerID="66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6" exitCode=0 Oct 30 01:23:09.645268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.645175 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerDied","Data":"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6"} Oct 30 01:23:09.645268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.645193 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" Oct 30 01:23:09.645268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.645216 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh" event={"ID":"6be1d1ce-3951-43dc-80d6-e72c9c86aee8","Type":"ContainerDied","Data":"e931a2bdec7dad447c6761cb7142c1c4dc8def18a37e3e1cb11012d88b237f69"} Oct 30 01:23:09.653046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.652966 5938 scope.go:122] "RemoveContainer" containerID="cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e" Oct 30 01:23:09.653209 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:09.653190 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e\": container with ID starting with cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e not found: ID does not exist" containerID="cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e" Oct 30 01:23:09.653254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.653216 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e"} err="failed to get container status \"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e\": rpc error: code = NotFound desc = could not find container \"cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e\": container with ID starting with cdbc16623100ee90284e1058537d76336bbb928a245bc05c58558b6bfce5359e not found: ID does not exist" Oct 30 01:23:09.653254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.653232 5938 scope.go:122] "RemoveContainer" containerID="66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6" Oct 30 01:23:09.665152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.665135 5938 scope.go:122] "RemoveContainer" containerID="0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924" Oct 30 01:23:09.675269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675241 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675293 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675350 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675644 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675479 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675700 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.675808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675746 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675774 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675838 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675878 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675906 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\") pod \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\" (UID: \"6be1d1ce-3951-43dc-80d6-e72c9c86aee8\") " Oct 30 01:23:09.675933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.675904 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.676322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676134 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.676500 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676440 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.676565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676505 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:09.676817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676790 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676816 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676831 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676845 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676874 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676890 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676904 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676917 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.676934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676931 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.677353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676944 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.677353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676957 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.677353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.676969 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48wr\" (UniqueName: \"kubernetes.io/projected/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b-kube-api-access-z48wr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.678016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.677925 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:09.678016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.677984 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.678170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.678132 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "kube-api-access-5nl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:09.678419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.678394 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy" pod "6be1d1ce-3951-43dc-80d6-e72c9c86aee8" (UID: "6be1d1ce-3951-43dc-80d6-e72c9c86aee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:09.683986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.683965 5938 scope.go:122] "RemoveContainer" containerID="66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6" Oct 30 01:23:09.684228 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:09.684212 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6\": container with ID starting with 66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6 not found: ID does not exist" containerID="66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6" Oct 30 01:23:09.684273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.684235 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6"} err="failed to get container status \"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6\": rpc error: code = NotFound desc = could not find container \"66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6\": container with ID starting with 66bc40b0e91cdd5ec5d19219bb6ff27c808b563210b547fcdcd9bbfefeb702d6 not found: ID does not exist" Oct 30 01:23:09.684273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.684251 5938 scope.go:122] "RemoveContainer" containerID="0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924" Oct 30 01:23:09.684476 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:09.684455 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924\": container with ID starting with 0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924 not found: ID does not exist" containerID="0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924" Oct 30 01:23:09.684659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.684483 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924"} err="failed to get container status \"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924\": rpc error: code = NotFound desc = could not find container \"0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924\": container with ID starting with 0ed6462ba8d7b8ee1770fb803d3f9722a1ce13ae2568962b4cd067cbf99d8924 not found: ID does not exist" Oct 30 01:23:09.704423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.704402 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:23:09.712232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.712209 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-nqdp4"] Oct 30 01:23:09.778151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.778098 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.778151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.778118 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.778151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.778127 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.778151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.778136 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nl88\" (UniqueName: \"kubernetes.io/projected/6be1d1ce-3951-43dc-80d6-e72c9c86aee8-kube-api-access-5nl88\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:09.876504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.876474 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" path="/var/lib/kubelet/pods/2160acac-590a-4dd3-bb22-23e6f85ffcb7/volumes" Oct 30 01:23:09.876934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.876915 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" path="/var/lib/kubelet/pods/5160a8f7-8d30-4938-81b6-7140cd82c4db/volumes" Oct 30 01:23:09.877362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.877348 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" path="/var/lib/kubelet/pods/c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b/volumes" Oct 30 01:23:09.989273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.989248 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:23:09.992482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:09.992459 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-d28kh"] Oct 30 01:23:10.259459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.259431 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5"] Oct 30 01:23:10.259984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.259967 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" Oct 30 01:23:10.259984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.259984 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.259998 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260007 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260022 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260027 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260039 5938 cpu_manager.go:414] "containerMap: removing container" podUID="69ea663f-75a1-4abb-bb7b-c94702adf2d4" containerName="revoke-keys" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260044 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ea663f-75a1-4abb-bb7b-c94702adf2d4" containerName="revoke-keys" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260054 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3f2ebccf-3746-4731-9935-efbc9f4ab00c" containerName="revoke-keys" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260059 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2ebccf-3746-4731-9935-efbc9f4ab00c" containerName="revoke-keys" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260083 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" Oct 30 01:23:10.260115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260092 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260180 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ea663f-75a1-4abb-bb7b-c94702adf2d4" containerName="revoke-keys" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260189 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2ebccf-3746-4731-9935-efbc9f4ab00c" containerName="revoke-keys" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260197 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45cf8d6-a8f6-4458-9f94-9b7e970a5b4b" containerName="istio-proxy" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260205 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260215 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2160acac-590a-4dd3-bb22-23e6f85ffcb7" containerName="main" Oct 30 01:23:10.260447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.260222 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5160a8f7-8d30-4938-81b6-7140cd82c4db" containerName="main" Oct 30 01:23:10.265161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.265143 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.285903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.281185 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:10.285903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.281731 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:10.285903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.281798 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:10.285903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.281833 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:10.286152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.285927 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5"] Oct 30 01:23:10.286152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.285987 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd"] Oct 30 01:23:10.286638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.286612 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.286638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.286636 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.286838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.286821 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" containerName="istio-proxy" Oct 30 01:23:10.290103 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.290085 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.310173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.310153 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd"] Oct 30 01:23:10.383027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.382990 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.383027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.383032 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.383225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.383062 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.383225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.383165 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrfs\" (UniqueName: \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.383225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.383201 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.383332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.383263 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgjb\" (UniqueName: \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.406177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.406142 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:10.475332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.475311 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:23:10.475510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.475491 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" podUID="671ec99d-4f3e-4ed1-aca5-01f510303092" containerName="payload-processing" containerID="cri-o://b39a76529b1df7a503687df64620c26251a3147665c6fa1aabaf9ba872919549" gracePeriod=30 Oct 30 01:23:10.484314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484288 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.484416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484329 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.484416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484372 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.484416 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484409 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrfs\" (UniqueName: \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.484597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484438 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.484597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.484520 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgjb\" (UniqueName: \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.485179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.485153 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.485299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.485184 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.486523 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.486499 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.486797 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.486776 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.495012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.494990 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:23:10.495238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.495197 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" podUID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" containerName="payload-pre-processing" containerID="cri-o://6b59345b321b418c58d585dd48de26c0a13b47917d0b28adfbac0aded005aa85" gracePeriod=30 Oct 30 01:23:10.508682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.508645 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrfs\" (UniqueName: \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\") pod \"maas-api-revoke-sub-584d0bc3-mmpcd\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.510291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.510233 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgjb\" (UniqueName: \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\") pod \"maas-api-revoke-sub-b3becc40-fd6q5\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.578668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.578643 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:10.597902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.597856 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:10.605721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.605698 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:10.667264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.659892 5938 generic.go:356] "Generic (PLEG): container finished" podID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" containerID="6b59345b321b418c58d585dd48de26c0a13b47917d0b28adfbac0aded005aa85" exitCode=0 Oct 30 01:23:10.667264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.660015 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" event={"ID":"0c5303b6-9f78-4d26-9ff3-2ff915f2422b","Type":"ContainerDied","Data":"6b59345b321b418c58d585dd48de26c0a13b47917d0b28adfbac0aded005aa85"} Oct 30 01:23:10.667264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.662815 5938 generic.go:356] "Generic (PLEG): container finished" podID="671ec99d-4f3e-4ed1-aca5-01f510303092" containerID="b39a76529b1df7a503687df64620c26251a3147665c6fa1aabaf9ba872919549" exitCode=0 Oct 30 01:23:10.667264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.663001 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" event={"ID":"671ec99d-4f3e-4ed1-aca5-01f510303092","Type":"ContainerDied","Data":"b39a76529b1df7a503687df64620c26251a3147665c6fa1aabaf9ba872919549"} Oct 30 01:23:10.741166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.741132 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:10.789730 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.789708 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:23:10.806319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.806300 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:23:10.891167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.891130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\") pod \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " Oct 30 01:23:10.891349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.891178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\" (UniqueName: \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\") pod \"671ec99d-4f3e-4ed1-aca5-01f510303092\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " Oct 30 01:23:10.891521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.891420 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\" (UniqueName: \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\") pod \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\" (UID: \"0c5303b6-9f78-4d26-9ff3-2ff915f2422b\") " Oct 30 01:23:10.891899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.891852 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\") pod \"671ec99d-4f3e-4ed1-aca5-01f510303092\" (UID: \"671ec99d-4f3e-4ed1-aca5-01f510303092\") " Oct 30 01:23:10.892285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.892254 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume" pod "0c5303b6-9f78-4d26-9ff3-2ff915f2422b" (UID: "0c5303b6-9f78-4d26-9ff3-2ff915f2422b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:10.892431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.892311 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume" pod "671ec99d-4f3e-4ed1-aca5-01f510303092" (UID: "671ec99d-4f3e-4ed1-aca5-01f510303092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:10.892532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.892487 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/671ec99d-4f3e-4ed1-aca5-01f510303092-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.892532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.892508 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.893908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.893847 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5"] Oct 30 01:23:10.896032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.895952 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f" pod "671ec99d-4f3e-4ed1-aca5-01f510303092" (UID: "671ec99d-4f3e-4ed1-aca5-01f510303092"). InnerVolumeSpecName "kube-api-access-8wv5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:10.896032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.895997 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg" pod "0c5303b6-9f78-4d26-9ff3-2ff915f2422b" (UID: "0c5303b6-9f78-4d26-9ff3-2ff915f2422b"). InnerVolumeSpecName "kube-api-access-7qkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:10.899116 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:10.899096 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ecbc2e_b8ab_4e40_8ba5_e58acde21371.slice/crio-f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378 WatchSource:0}: Error finding container f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378: Status 404 returned error can't find the container with id f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378 Oct 30 01:23:10.938205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.938184 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd"] Oct 30 01:23:10.939581 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:10.939558 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9397cf03_fcfd_42b2_ac44_e17e36230e9b.slice/crio-cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9 WatchSource:0}: Error finding container cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9: Status 404 returned error can't find the container with id cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9 Oct 30 01:23:10.993346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.993323 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wv5f\" (UniqueName: \"kubernetes.io/projected/671ec99d-4f3e-4ed1-aca5-01f510303092-kube-api-access-8wv5f\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:10.993346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:10.993347 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qkgg\" (UniqueName: \"kubernetes.io/projected/0c5303b6-9f78-4d26-9ff3-2ff915f2422b-kube-api-access-7qkgg\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.004016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.003994 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.088831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.088774 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn"] Oct 30 01:23:11.090940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.090917 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-658c99-r4wqn"] Oct 30 01:23:11.120790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.120756 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" containerID="cri-o://8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817" gracePeriod=45 Oct 30 01:23:11.135649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.135585 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" containerID="cri-o://00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda" gracePeriod=45 Oct 30 01:23:11.135649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.135617 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" containerID="cri-o://f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c" gracePeriod=45 Oct 30 01:23:11.151016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.150995 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.172297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.172268 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused" Oct 30 01:23:11.387515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.387485 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 01:23:11.500732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500702 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.500854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500755 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.500854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500788 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.500854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500827 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.501125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500962 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\" (UniqueName: \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.501125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.500971 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.501125 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501028 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\") pod \"d3a05680-d6ad-401e-ac93-92248c1f3886\" (UID: \"d3a05680-d6ad-401e-ac93-92248c1f3886\") " Oct 30 01:23:11.501302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501161 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.501368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.501954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501493 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.501954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501512 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.501954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.501529 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.503462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.503432 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "kube-api-access-5x28q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:11.503571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.503494 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:11.503817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.503790 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm" pod "d3a05680-d6ad-401e-ac93-92248c1f3886" (UID: "d3a05680-d6ad-401e-ac93-92248c1f3886"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.510364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.510338 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 01:23:11.526342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.526322 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 01:23:11.602443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602406 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.602641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602492 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.602641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602524 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.602641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602573 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.602641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602579 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.602901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\" (UniqueName: \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.602901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.602713 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603604 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603114 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603645 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\" (UniqueName: \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603116 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603681 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\") pod \"641fb22e-6646-4270-a40c-8515fe2f8e4b\" (UID: \"641fb22e-6646-4270-a40c-8515fe2f8e4b\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603757 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.604016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.603838 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\") pod \"f984a109-45ff-4560-8ed4-47457f5b29e6\" (UID: \"f984a109-45ff-4560-8ed4-47457f5b29e6\") " Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604260 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x28q\" (UniqueName: \"kubernetes.io/projected/d3a05680-d6ad-401e-ac93-92248c1f3886-kube-api-access-5x28q\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604276 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604283 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604321 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a05680-d6ad-401e-ac93-92248c1f3886-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604335 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a05680-d6ad-401e-ac93-92248c1f3886-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604348 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604360 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604489 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.604560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.604506 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.605264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.605226 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:11.605617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.605584 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:11.606592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.606566 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "kube-api-access-pcjqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:11.607015 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.606935 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "kube-api-access-h2rbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:11.607090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.607038 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm" pod "641fb22e-6646-4270-a40c-8515fe2f8e4b" (UID: "641fb22e-6646-4270-a40c-8515fe2f8e4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.607722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.607697 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm" pod "f984a109-45ff-4560-8ed4-47457f5b29e6" (UID: "f984a109-45ff-4560-8ed4-47457f5b29e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:11.612330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.612246 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.675555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.675525 5938 generic.go:356] "Generic (PLEG): container finished" podID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerID="8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817" exitCode=0 Oct 30 01:23:11.676047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.675578 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" Oct 30 01:23:11.676047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.675609 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" event={"ID":"641fb22e-6646-4270-a40c-8515fe2f8e4b","Type":"ContainerDied","Data":"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817"} Oct 30 01:23:11.676047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.675651 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5" event={"ID":"641fb22e-6646-4270-a40c-8515fe2f8e4b","Type":"ContainerDied","Data":"03355d3851a4aaea1e0c3028c32b93ce9a15f7d49538497d4aad2f0d19591168"} Oct 30 01:23:11.676047 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.675668 5938 scope.go:122] "RemoveContainer" containerID="8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817" Oct 30 01:23:11.677415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.677386 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" event={"ID":"9397cf03-fcfd-42b2-ac44-e17e36230e9b","Type":"ContainerStarted","Data":"990b80272e9f128a63d2d72b5026067e01788d85f5041441530874411cf3a83d"} Oct 30 01:23:11.677538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.677446 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" event={"ID":"9397cf03-fcfd-42b2-ac44-e17e36230e9b","Type":"ContainerStarted","Data":"cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9"} Oct 30 01:23:11.679013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.678998 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" Oct 30 01:23:11.679117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.679051 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg" event={"ID":"0c5303b6-9f78-4d26-9ff3-2ff915f2422b","Type":"ContainerDied","Data":"dc75f47c455627018f1e81370e95ca82b1832332527f93ffb7a11122115d6271"} Oct 30 01:23:11.682671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.682650 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" event={"ID":"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371","Type":"ContainerStarted","Data":"cd434a1120310d13926eaee15770f7c0beda2d9263b3da6ba31318778fef3e19"} Oct 30 01:23:11.682752 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.682694 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" event={"ID":"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371","Type":"ContainerStarted","Data":"f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378"} Oct 30 01:23:11.684488 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.684459 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" event={"ID":"671ec99d-4f3e-4ed1-aca5-01f510303092","Type":"ContainerDied","Data":"b9a2aedb523858e74dd7516006688078456c7261dcd19d7882ef4a46e056e456"} Oct 30 01:23:11.684604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.684559 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt" Oct 30 01:23:11.686758 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.686717 5938 generic.go:356] "Generic (PLEG): container finished" podID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerID="00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda" exitCode=0 Oct 30 01:23:11.686874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.686758 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" Oct 30 01:23:11.686874 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.686814 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" event={"ID":"d3a05680-d6ad-401e-ac93-92248c1f3886","Type":"ContainerDied","Data":"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda"} Oct 30 01:23:11.687132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.686876 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm" event={"ID":"d3a05680-d6ad-401e-ac93-92248c1f3886","Type":"ContainerDied","Data":"addd2a9c8e6466a168e2218cfe00e0be8211490915bb232235760f44f7971793"} Oct 30 01:23:11.687618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.687599 5938 scope.go:122] "RemoveContainer" containerID="8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817" Oct 30 01:23:11.688469 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:11.688438 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817\": container with ID starting with 8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817 not found: ID does not exist" containerID="8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817" Oct 30 01:23:11.688561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.688471 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817"} err="failed to get container status \"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817\": rpc error: code = NotFound desc = could not find container \"8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817\": container with ID starting with 8bc124169d2e6a670c1406e3c385de75ee45ff54ec82d1eb94b2b09483106817 not found: ID does not exist" Oct 30 01:23:11.688561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.688497 5938 scope.go:122] "RemoveContainer" containerID="6b59345b321b418c58d585dd48de26c0a13b47917d0b28adfbac0aded005aa85" Oct 30 01:23:11.689737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.689590 5938 generic.go:356] "Generic (PLEG): container finished" podID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerID="f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c" exitCode=0 Oct 30 01:23:11.689737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.689636 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" event={"ID":"f984a109-45ff-4560-8ed4-47457f5b29e6","Type":"ContainerDied","Data":"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c"} Oct 30 01:23:11.689737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.689661 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" event={"ID":"f984a109-45ff-4560-8ed4-47457f5b29e6","Type":"ContainerDied","Data":"0c24499999862ea32c82cc698b015ede02b2fef32a7bc24698090336b12a6dda"} Oct 30 01:23:11.690429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.689802 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7" Oct 30 01:23:11.700688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.700659 5938 scope.go:122] "RemoveContainer" containerID="b39a76529b1df7a503687df64620c26251a3147665c6fa1aabaf9ba872919549" Oct 30 01:23:11.705328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705309 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705329 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705338 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f984a109-45ff-4560-8ed4-47457f5b29e6-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705347 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641fb22e-6646-4270-a40c-8515fe2f8e4b-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705356 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcjqd\" (UniqueName: \"kubernetes.io/projected/641fb22e-6646-4270-a40c-8515fe2f8e4b-kube-api-access-pcjqd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705364 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705373 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rbb\" (UniqueName: \"kubernetes.io/projected/f984a109-45ff-4560-8ed4-47457f5b29e6-kube-api-access-h2rbb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705381 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641fb22e-6646-4270-a40c-8515fe2f8e4b-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.705489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.705388 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f984a109-45ff-4560-8ed4-47457f5b29e6-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:11.711473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.711457 5938 scope.go:122] "RemoveContainer" containerID="00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda" Oct 30 01:23:11.719366 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.719348 5938 scope.go:122] "RemoveContainer" containerID="00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda" Oct 30 01:23:11.719624 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:11.719607 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda\": container with ID starting with 00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda not found: ID does not exist" containerID="00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda" Oct 30 01:23:11.719667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.719631 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda"} err="failed to get container status \"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda\": rpc error: code = NotFound desc = could not find container \"00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda\": container with ID starting with 00b5540ed462da124ce4cbae9b9f986ce8863773612adc9198c43955af111cda not found: ID does not exist" Oct 30 01:23:11.719667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.719652 5938 scope.go:122] "RemoveContainer" containerID="f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c" Oct 30 01:23:11.727201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.727185 5938 scope.go:122] "RemoveContainer" containerID="f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c" Oct 30 01:23:11.727462 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:11.727441 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c\": container with ID starting with f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c not found: ID does not exist" containerID="f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c" Oct 30 01:23:11.727509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.727470 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c"} err="failed to get container status \"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c\": rpc error: code = NotFound desc = could not find container \"f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c\": container with ID starting with f94d722a87b1ff535b0eda82101429feafae53576d9c268ecb2d6f22b9c1136c not found: ID does not exist" Oct 30 01:23:11.756177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.756157 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 01:23:11.760191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.760171 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/distinct-w4-ca5363-kserve-6fd77d7b86-xzkd5"] Oct 30 01:23:11.809903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.809843 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" podStartSLOduration=1.809829669 podStartE2EDuration="1.809829669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:11.809483058 +0000 UTC m=+3044.049718172" watchObservedRunningTime="2026-10-30 01:23:11.809829669 +0000 UTC m=+3044.050064779" Oct 30 01:23:11.885291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.885265 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" path="/var/lib/kubelet/pods/641fb22e-6646-4270-a40c-8515fe2f8e4b/volumes" Oct 30 01:23:11.885795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.885777 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be1d1ce-3951-43dc-80d6-e72c9c86aee8" path="/var/lib/kubelet/pods/6be1d1ce-3951-43dc-80d6-e72c9c86aee8/volumes" Oct 30 01:23:11.886464 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.886446 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7" path="/var/lib/kubelet/pods/a48b8b7c-cd56-48a7-8285-c28ff7a4bfd7/volumes" Oct 30 01:23:11.902635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.902590 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" podStartSLOduration=1.90257853 podStartE2EDuration="1.90257853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:11.867139404 +0000 UTC m=+3044.107374531" watchObservedRunningTime="2026-10-30 01:23:11.90257853 +0000 UTC m=+3044.142813650" Oct 30 01:23:11.903487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.903464 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:23:11.906443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.906421 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-658c99-8578c5b85b-z8slt"] Oct 30 01:23:11.938243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.938214 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 01:23:11.943162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.943140 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/simulator-w4-ca5363-kserve-6547b69dc8-pd6v7"] Oct 30 01:23:11.974836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.974814 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 01:23:11.980986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:11.980963 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca5363/premium-w4-ca5363-kserve-7cb6dcd795-62srm"] Oct 30 01:23:12.014106 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.014086 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:23:12.019697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.019677 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-658c99-5dd6d56c55-gn8lg"] Oct 30 01:23:12.191577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.191514 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" containerID="cri-o://b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e" gracePeriod=25 Oct 30 01:23:12.697176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.697146 5938 generic.go:356] "Generic (PLEG): container finished" podID="e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" containerID="cd434a1120310d13926eaee15770f7c0beda2d9263b3da6ba31318778fef3e19" exitCode=0 Oct 30 01:23:12.697539 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.697235 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" event={"ID":"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371","Type":"ContainerDied","Data":"cd434a1120310d13926eaee15770f7c0beda2d9263b3da6ba31318778fef3e19"} Oct 30 01:23:12.702320 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.702299 5938 generic.go:356] "Generic (PLEG): container finished" podID="9397cf03-fcfd-42b2-ac44-e17e36230e9b" containerID="990b80272e9f128a63d2d72b5026067e01788d85f5041441530874411cf3a83d" exitCode=0 Oct 30 01:23:12.702444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.702379 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" event={"ID":"9397cf03-fcfd-42b2-ac44-e17e36230e9b","Type":"ContainerDied","Data":"990b80272e9f128a63d2d72b5026067e01788d85f5041441530874411cf3a83d"} Oct 30 01:23:12.891091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:12.891057 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:13.274582 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.274512 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:13.849152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.849122 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:13.871341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.871318 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:13.874614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.874587 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" path="/var/lib/kubelet/pods/0c5303b6-9f78-4d26-9ff3-2ff915f2422b/volumes" Oct 30 01:23:13.875130 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.875109 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="671ec99d-4f3e-4ed1-aca5-01f510303092" path="/var/lib/kubelet/pods/671ec99d-4f3e-4ed1-aca5-01f510303092/volumes" Oct 30 01:23:13.875587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.875568 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" path="/var/lib/kubelet/pods/d3a05680-d6ad-401e-ac93-92248c1f3886/volumes" Oct 30 01:23:13.876332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.876311 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" path="/var/lib/kubelet/pods/f984a109-45ff-4560-8ed4-47457f5b29e6/volumes" Oct 30 01:23:13.926176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926114 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\" (UniqueName: \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\") pod \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " Oct 30 01:23:13.926321 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926197 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\") pod \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " Oct 30 01:23:13.926427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926401 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\") pod \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " Oct 30 01:23:13.926575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926465 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\" (UniqueName: \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\") pod \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\" (UID: \"9397cf03-fcfd-42b2-ac44-e17e36230e9b\") " Oct 30 01:23:13.926575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926529 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\") pod \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " Oct 30 01:23:13.927389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926672 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\") pod \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\" (UID: \"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371\") " Oct 30 01:23:13.927389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.926778 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca" pod "9397cf03-fcfd-42b2-ac44-e17e36230e9b" (UID: "9397cf03-fcfd-42b2-ac44-e17e36230e9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:13.927389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.927218 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:13.927571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.927459 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca" pod "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" (UID: "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:13.929410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.929381 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb" pod "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" (UID: "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371"). InnerVolumeSpecName "kube-api-access-wcgjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:13.931157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.931119 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca" pod "9397cf03-fcfd-42b2-ac44-e17e36230e9b" (UID: "9397cf03-fcfd-42b2-ac44-e17e36230e9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:13.931726 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.931699 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca" pod "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" (UID: "e6ecbc2e-b8ab-4e40-8ba5-e58acde21371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:13.932186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:13.932158 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs" pod "9397cf03-fcfd-42b2-ac44-e17e36230e9b" (UID: "9397cf03-fcfd-42b2-ac44-e17e36230e9b"). InnerVolumeSpecName "kube-api-access-ccrfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:14.028149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.028072 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgjb\" (UniqueName: \"kubernetes.io/projected/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-kube-api-access-wcgjb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:14.028149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.028108 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9397cf03-fcfd-42b2-ac44-e17e36230e9b-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:14.028149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.028122 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrfs\" (UniqueName: \"kubernetes.io/projected/9397cf03-fcfd-42b2-ac44-e17e36230e9b-kube-api-access-ccrfs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:14.028149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.028136 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:14.028149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.028151 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:14.717728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.717702 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" Oct 30 01:23:14.717940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.717700 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd" event={"ID":"9397cf03-fcfd-42b2-ac44-e17e36230e9b","Type":"ContainerDied","Data":"cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9"} Oct 30 01:23:14.717940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.717827 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cccf576401e7303deefa512d996ac311641963a260e15aeeb2af3be9721382b9" Oct 30 01:23:14.720012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.719981 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" event={"ID":"e6ecbc2e-b8ab-4e40-8ba5-e58acde21371","Type":"ContainerDied","Data":"f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378"} Oct 30 01:23:14.720124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.720021 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5" Oct 30 01:23:14.720124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.720039 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c640e95547503e8bf4d78425d3f7effa7f0a2d67cfadebbcc6b25fa3ae7378" Oct 30 01:23:14.788129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:14.788101 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.269685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.269654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.375819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.375786 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb"] Oct 30 01:23:15.387114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.387087 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cefb27-8zrdb"] Oct 30 01:23:15.424811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.424777 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.876022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.875989 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15893fad-3979-4219-a2df-bbe1bb845cf1" path="/var/lib/kubelet/pods/15893fad-3979-4219-a2df-bbe1bb845cf1/volumes" Oct 30 01:23:15.879565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:15.879521 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.119445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:17.119414 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.897552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:17.897523 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.046141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.046102 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.230098 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.229988 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.680057 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680025 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq"] Oct 30 01:23:18.680471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680457 5938 cpu_manager.go:414] "containerMap: removing container" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680472 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680489 5938 cpu_manager.go:414] "containerMap: removing container" podUID="671ec99d-4f3e-4ed1-aca5-01f510303092" containerName="payload-processing" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680495 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="671ec99d-4f3e-4ed1-aca5-01f510303092" containerName="payload-processing" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680514 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680519 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" Oct 30 01:23:18.680534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680531 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" containerName="revoke-keys" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680551 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" containerName="revoke-keys" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680565 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" containerName="payload-pre-processing" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680571 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" containerName="payload-pre-processing" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680594 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9397cf03-fcfd-42b2-ac44-e17e36230e9b" containerName="revoke-keys" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680600 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9397cf03-fcfd-42b2-ac44-e17e36230e9b" containerName="revoke-keys" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680611 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680615 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680678 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5303b6-9f78-4d26-9ff3-2ff915f2422b" containerName="payload-pre-processing" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680689 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9397cf03-fcfd-42b2-ac44-e17e36230e9b" containerName="revoke-keys" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680697 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a05680-d6ad-401e-ac93-92248c1f3886" containerName="main" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680703 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="641fb22e-6646-4270-a40c-8515fe2f8e4b" containerName="main" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680709 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="671ec99d-4f3e-4ed1-aca5-01f510303092" containerName="payload-processing" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680716 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f984a109-45ff-4560-8ed4-47457f5b29e6" containerName="main" Oct 30 01:23:18.680718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.680723 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" containerName="revoke-keys" Oct 30 01:23:18.685560 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.685544 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.701978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.701953 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq"] Oct 30 01:23:18.713897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.712633 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:18.713897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.712667 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:18.713897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.712689 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:18.713897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.712824 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:18.784051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.784021 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.784188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.784078 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.784270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.784219 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbx4\" (UniqueName: \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.885470 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.885442 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbx4\" (UniqueName: \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.885589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.885559 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.885672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.885609 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.886309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.886280 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.887686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.887667 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.907484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.907454 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbx4\" (UniqueName: \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:18.935727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.935664 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.998790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:18.998757 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:19.294251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:19.294217 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq"] Oct 30 01:23:19.297309 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:19.297282 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe196918_4650_46b7_85d6_a46c08f1bc48.slice/crio-debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0 WatchSource:0}: Error finding container debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0: Status 404 returned error can't find the container with id debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0 Oct 30 01:23:19.753127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:19.753095 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" event={"ID":"fe196918-4650-46b7-85d6-a46c08f1bc48","Type":"ContainerStarted","Data":"953199c0b005ddc4c9704591a7986eacfb6965898b37658c7759801d9cf90014"} Oct 30 01:23:19.753300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:19.753153 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" event={"ID":"fe196918-4650-46b7-85d6-a46c08f1bc48","Type":"ContainerStarted","Data":"debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0"} Oct 30 01:23:19.792252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:19.792209 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" podStartSLOduration=1.792197356 podStartE2EDuration="1.792197356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:19.791002802 +0000 UTC m=+3052.031237916" watchObservedRunningTime="2026-10-30 01:23:19.792197356 +0000 UTC m=+3052.032432493" Oct 30 01:23:20.631310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:20.631282 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.762619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:20.762587 5938 generic.go:356] "Generic (PLEG): container finished" podID="fe196918-4650-46b7-85d6-a46c08f1bc48" containerID="953199c0b005ddc4c9704591a7986eacfb6965898b37658c7759801d9cf90014" exitCode=0 Oct 30 01:23:20.762834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:20.762641 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" event={"ID":"fe196918-4650-46b7-85d6-a46c08f1bc48","Type":"ContainerDied","Data":"953199c0b005ddc4c9704591a7986eacfb6965898b37658c7759801d9cf90014"} Oct 30 01:23:21.169925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.169848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.275232 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.275204 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng"] Oct 30 01:23:21.277958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.277935 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64dc0-j2cng"] Oct 30 01:23:21.309537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.309515 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.449770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.449698 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.782766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.782726 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.875249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.875213 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270a2d2a-8be6-46e8-9329-7052e823986c" path="/var/lib/kubelet/pods/270a2d2a-8be6-46e8-9329-7052e823986c/volumes" Oct 30 01:23:21.917501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:21.917480 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:22.014123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.014100 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\" (UniqueName: \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\") pod \"fe196918-4650-46b7-85d6-a46c08f1bc48\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " Oct 30 01:23:22.014270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.014173 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\") pod \"fe196918-4650-46b7-85d6-a46c08f1bc48\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " Oct 30 01:23:22.014270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.014232 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\") pod \"fe196918-4650-46b7-85d6-a46c08f1bc48\" (UID: \"fe196918-4650-46b7-85d6-a46c08f1bc48\") " Oct 30 01:23:22.014643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.014620 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca" pod "fe196918-4650-46b7-85d6-a46c08f1bc48" (UID: "fe196918-4650-46b7-85d6-a46c08f1bc48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:22.016219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.016193 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca" pod "fe196918-4650-46b7-85d6-a46c08f1bc48" (UID: "fe196918-4650-46b7-85d6-a46c08f1bc48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:22.016306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.016245 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4" pod "fe196918-4650-46b7-85d6-a46c08f1bc48" (UID: "fe196918-4650-46b7-85d6-a46c08f1bc48"). InnerVolumeSpecName "kube-api-access-cmbx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:22.114960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.114936 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:22.114960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.114959 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbx4\" (UniqueName: \"kubernetes.io/projected/fe196918-4650-46b7-85d6-a46c08f1bc48-kube-api-access-cmbx4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:22.115093 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.114969 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe196918-4650-46b7-85d6-a46c08f1bc48-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:22.422307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.422270 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.570007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.569966 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.717235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.717148 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 01:23:22.717677 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.717643 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" containerID="cri-o://740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b" gracePeriod=30 Oct 30 01:23:22.777595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.777565 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" Oct 30 01:23:22.777790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.777597 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq" event={"ID":"fe196918-4650-46b7-85d6-a46c08f1bc48","Type":"ContainerDied","Data":"debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0"} Oct 30 01:23:22.777790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.777653 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="debe84e17f90b209398c711e1caf525420ac9a9616c1474ff08e63f5dd5206e0" Oct 30 01:23:22.793913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.793886 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.891331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:22.891295 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:23.187043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.187005 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:23.737934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.737902 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:23.844935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.844895 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z"] Oct 30 01:23:23.855882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.855834 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca5363-6f97z"] Oct 30 01:23:23.874770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.874738 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:23.875385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:23.875367 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4396ac85-318b-49f3-bd86-7d9ed463d495" path="/var/lib/kubelet/pods/4396ac85-318b-49f3-bd86-7d9ed463d495/volumes" Oct 30 01:23:24.010554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:24.010458 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.371847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:24.371798 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.730121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:25.730080 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:27.584281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:27.584249 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:28.160148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.160117 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:28.296910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.296856 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:28.446088 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.446008 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:23:28.446341 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.446295 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" containerID="cri-o://b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00" gracePeriod=30 Oct 30 01:23:28.599666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.599633 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:28.784830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.784804 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 01:23:28.815069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.815036 5938 generic.go:356] "Generic (PLEG): container finished" podID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerID="740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b" exitCode=0 Oct 30 01:23:28.815177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.815089 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" Oct 30 01:23:28.815177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.815116 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerDied","Data":"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b"} Oct 30 01:23:28.815177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.815168 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b" event={"ID":"2e48bff3-a057-4901-bb9a-8efa404e8ae9","Type":"ContainerDied","Data":"6564a9d3b2fc036a539df61cbc99a11e74d8fd07089a53e39b1806357289af1d"} Oct 30 01:23:28.815300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.815190 5938 scope.go:122] "RemoveContainer" containerID="740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b" Oct 30 01:23:28.824058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.824039 5938 scope.go:122] "RemoveContainer" containerID="ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162" Oct 30 01:23:28.835190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.835081 5938 scope.go:122] "RemoveContainer" containerID="740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b" Oct 30 01:23:28.835436 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:28.835411 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b\": container with ID starting with 740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b not found: ID does not exist" containerID="740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b" Oct 30 01:23:28.835515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.835447 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b"} err="failed to get container status \"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b\": rpc error: code = NotFound desc = could not find container \"740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b\": container with ID starting with 740983d998d5988f1516e973a7f751a74a0558fa5165d48c01899b5f8ea54c5b not found: ID does not exist" Oct 30 01:23:28.835515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.835468 5938 scope.go:122] "RemoveContainer" containerID="ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162" Oct 30 01:23:28.835793 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:28.835772 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162\": container with ID starting with ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162 not found: ID does not exist" containerID="ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162" Oct 30 01:23:28.835917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.835802 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162"} err="failed to get container status \"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162\": rpc error: code = NotFound desc = could not find container \"ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162\": container with ID starting with ae67c21063bd8aa224e1da054bd3e56c89381932ca4ff3791b3a6c525bdab162 not found: ID does not exist" Oct 30 01:23:28.838039 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:28.838012 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:23:28.885257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885226 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:28.885529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.884921 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885570 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885611 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885648 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885687 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885705 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885760 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885785 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.885908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.885836 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\") pod \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\" (UID: \"2e48bff3-a057-4901-bb9a-8efa404e8ae9\") " Oct 30 01:23:28.886139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.886108 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:28.886251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.886176 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.886766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.886550 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:28.886766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.886714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:28.886967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.886799 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:28.890592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.890563 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:28.891950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.891887 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:28.892092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.892056 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "kube-api-access-qq4g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:28.892258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.892234 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token" pod "2e48bff3-a057-4901-bb9a-8efa404e8ae9" (UID: "2e48bff3-a057-4901-bb9a-8efa404e8ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:28.958254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958199 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4"] Oct 30 01:23:28.958621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958609 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:28.958669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958623 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:28.958669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958639 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:28.958669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958644 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:28.958669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958660 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fe196918-4650-46b7-85d6-a46c08f1bc48" containerName="revoke-keys" Oct 30 01:23:28.958669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958665 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe196918-4650-46b7-85d6-a46c08f1bc48" containerName="revoke-keys" Oct 30 01:23:28.958824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958742 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:28.958824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.958753 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe196918-4650-46b7-85d6-a46c08f1bc48" containerName="revoke-keys" Oct 30 01:23:28.963264 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.963246 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:28.977801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.977777 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:28.977930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.977878 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:28.980481 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.980459 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4"] Oct 30 01:23:28.987479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987456 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:28.987585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987506 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d85c\" (UniqueName: \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:28.987647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987614 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:28.987763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987743 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.987828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987768 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.987828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987783 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.987949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987923 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.987949 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987945 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e48bff3-a057-4901-bb9a-8efa404e8ae9-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.988011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987960 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.988011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987975 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e48bff3-a057-4901-bb9a-8efa404e8ae9-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.988011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.987992 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq4g7\" (UniqueName: \"kubernetes.io/projected/2e48bff3-a057-4901-bb9a-8efa404e8ae9-kube-api-access-qq4g7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:28.996315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.996276 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:28.996417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.996344 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:28.996487 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:28.996306 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.088431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.088406 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.088602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.088443 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d85c\" (UniqueName: \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.088602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.088471 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.089137 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.089114 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.090710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.090691 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.110853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.110831 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d85c\" (UniqueName: \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.162151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.162120 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 01:23:29.164050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.164029 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-bbb7b"] Oct 30 01:23:29.276397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.276329 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:29.294611 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.294587 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.442195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.442166 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.572388 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.572304 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4"] Oct 30 01:23:29.580175 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:29.580145 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd93939_b863_4d83_91a7_03ed30bdf7a0.slice/crio-f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa WatchSource:0}: Error finding container f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa: Status 404 returned error can't find the container with id f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa Oct 30 01:23:29.604422 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.604400 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.747832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.747798 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.826317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.826241 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" event={"ID":"0fd93939-b863-4d83-91a7-03ed30bdf7a0","Type":"ContainerStarted","Data":"f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa"} Oct 30 01:23:29.875070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.875042 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" path="/var/lib/kubelet/pods/2e48bff3-a057-4901-bb9a-8efa404e8ae9/volumes" Oct 30 01:23:29.925171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:29.925142 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:30.834557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:30.834525 5938 generic.go:356] "Generic (PLEG): container finished" podID="0fd93939-b863-4d83-91a7-03ed30bdf7a0" containerID="7482387ff4866fad23510ba33f18c6e8913fdc3e26a92043890f8e591424a237" exitCode=0 Oct 30 01:23:30.834968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:30.834612 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" event={"ID":"0fd93939-b863-4d83-91a7-03ed30bdf7a0","Type":"ContainerDied","Data":"7482387ff4866fad23510ba33f18c6e8913fdc3e26a92043890f8e591424a237"} Oct 30 01:23:30.975928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:30.975804 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:31.546763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.546728 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:31.622052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.622016 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq"] Oct 30 01:23:31.633738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.633709 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-009e78-wpkhq"] Oct 30 01:23:31.695750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.695716 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:31.875985 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.875956 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe196918-4650-46b7-85d6-a46c08f1bc48" path="/var/lib/kubelet/pods/fe196918-4650-46b7-85d6-a46c08f1bc48/volumes" Oct 30 01:23:31.992635 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:31.992606 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:32.002314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.002295 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:32.112193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.112166 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\") pod \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " Oct 30 01:23:32.112403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.112212 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\" (UniqueName: \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\") pod \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " Oct 30 01:23:32.112403 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.112297 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\") pod \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\" (UID: \"0fd93939-b863-4d83-91a7-03ed30bdf7a0\") " Oct 30 01:23:32.112555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.112531 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca" pod "0fd93939-b863-4d83-91a7-03ed30bdf7a0" (UID: "0fd93939-b863-4d83-91a7-03ed30bdf7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:32.112682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.112665 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.114451 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.114421 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca" pod "0fd93939-b863-4d83-91a7-03ed30bdf7a0" (UID: "0fd93939-b863-4d83-91a7-03ed30bdf7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:32.114576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.114554 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c" pod "0fd93939-b863-4d83-91a7-03ed30bdf7a0" (UID: "0fd93939-b863-4d83-91a7-03ed30bdf7a0"). InnerVolumeSpecName "kube-api-access-5d85c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:32.214070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.214044 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d85c\" (UniqueName: \"kubernetes.io/projected/0fd93939-b863-4d83-91a7-03ed30bdf7a0-kube-api-access-5d85c\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.214156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.214071 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd93939-b863-4d83-91a7-03ed30bdf7a0-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.464044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.464015 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:23:32.618557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.618520 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\") pod \"5e6823d9-fa04-412f-940e-97018cb6d90a\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " Oct 30 01:23:32.618726 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.618578 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\" (UniqueName: \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\") pod \"5e6823d9-fa04-412f-940e-97018cb6d90a\" (UID: \"5e6823d9-fa04-412f-940e-97018cb6d90a\") " Oct 30 01:23:32.619054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.619024 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume" pod "5e6823d9-fa04-412f-940e-97018cb6d90a" (UID: "5e6823d9-fa04-412f-940e-97018cb6d90a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:32.620719 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.620692 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7" pod "5e6823d9-fa04-412f-940e-97018cb6d90a" (UID: "5e6823d9-fa04-412f-940e-97018cb6d90a"). InnerVolumeSpecName "kube-api-access-kcdb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:32.720435 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.720356 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e6823d9-fa04-412f-940e-97018cb6d90a-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.720435 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.720395 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdb7\" (UniqueName: \"kubernetes.io/projected/5e6823d9-fa04-412f-940e-97018cb6d90a-kube-api-access-kcdb7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:32.848842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.848813 5938 generic.go:356] "Generic (PLEG): container finished" podID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerID="b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e" exitCode=0 Oct 30 01:23:32.849042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.848877 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" Oct 30 01:23:32.849042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.848909 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" event={"ID":"5e6823d9-fa04-412f-940e-97018cb6d90a","Type":"ContainerDied","Data":"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e"} Oct 30 01:23:32.849042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.848998 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp" event={"ID":"5e6823d9-fa04-412f-940e-97018cb6d90a","Type":"ContainerDied","Data":"080f71010e305c3f42c01039a0b8a152ec4d4d3be797568e7d3a5263f99cddd4"} Oct 30 01:23:32.849042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.849028 5938 scope.go:122] "RemoveContainer" containerID="b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e" Oct 30 01:23:32.850362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.850311 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" Oct 30 01:23:32.850362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.850324 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4" event={"ID":"0fd93939-b863-4d83-91a7-03ed30bdf7a0","Type":"ContainerDied","Data":"f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa"} Oct 30 01:23:32.850458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.850368 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3763646e23073c2ef02089af5f465189eb54543455003a8baf9f9f0da8039aa" Oct 30 01:23:32.858903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.858887 5938 scope.go:122] "RemoveContainer" containerID="b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e" Oct 30 01:23:32.859192 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:32.859171 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e\": container with ID starting with b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e not found: ID does not exist" containerID="b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e" Oct 30 01:23:32.859275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.859203 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e"} err="failed to get container status \"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e\": rpc error: code = NotFound desc = could not find container \"b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e\": container with ID starting with b9ab3c68b0da2b079629c952dc1f7bdb76dd6fc4cebfaa336b2049ece503b48e not found: ID does not exist" Oct 30 01:23:32.911083 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.911058 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:23:32.918398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:32.918379 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64dc0-8555d77c9d-sxwmp"] Oct 30 01:23:33.481039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.481012 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:33.703612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.703590 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:23:33.729157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729097 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729136 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729166 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729208 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729279 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729343 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729369 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.729436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729400 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729451 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729484 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\") pod \"9452a925-9485-4073-bc8c-bf81a4d79275\" (UID: \"9452a925-9485-4073-bc8c-bf81a4d79275\") " Oct 30 01:23:33.729615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729518 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.729615 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729534 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.729882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729844 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.731128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.729989 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.731128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.730011 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.731128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.730077 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.731128 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.730159 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:33.731484 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.731464 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:33.731682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.731658 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:33.731781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.731766 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "kube-api-access-mcxn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:33.731924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.731906 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo" pod "9452a925-9485-4073-bc8c-bf81a4d79275" (UID: "9452a925-9485-4073-bc8c-bf81a4d79275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:33.830587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830557 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9452a925-9485-4073-bc8c-bf81a4d79275-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.830587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830585 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.830587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830593 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9452a925-9485-4073-bc8c-bf81a4d79275-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.830788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830602 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9452a925-9485-4073-bc8c-bf81a4d79275-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.830788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830610 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.830788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.830618 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxn9\" (UniqueName: \"kubernetes.io/projected/9452a925-9485-4073-bc8c-bf81a4d79275-kube-api-access-mcxn9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:33.859323 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.859294 5938 generic.go:356] "Generic (PLEG): container finished" podID="9452a925-9485-4073-bc8c-bf81a4d79275" containerID="b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00" exitCode=0 Oct 30 01:23:33.859475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.859344 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerDied","Data":"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00"} Oct 30 01:23:33.859475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.859350 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" Oct 30 01:23:33.859475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.859396 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs" event={"ID":"9452a925-9485-4073-bc8c-bf81a4d79275","Type":"ContainerDied","Data":"c980fb5fea58df163d98ffb08ad857d1d82e80f47614372a141950a5844f2718"} Oct 30 01:23:33.859475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.859428 5938 scope.go:122] "RemoveContainer" containerID="b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00" Oct 30 01:23:33.873193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.873169 5938 scope.go:122] "RemoveContainer" containerID="e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db" Oct 30 01:23:33.877096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.877071 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" path="/var/lib/kubelet/pods/5e6823d9-fa04-412f-940e-97018cb6d90a/volumes" Oct 30 01:23:33.882736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.882718 5938 scope.go:122] "RemoveContainer" containerID="b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00" Oct 30 01:23:33.883007 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:33.882989 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00\": container with ID starting with b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00 not found: ID does not exist" containerID="b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00" Oct 30 01:23:33.883055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.883017 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00"} err="failed to get container status \"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00\": rpc error: code = NotFound desc = could not find container \"b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00\": container with ID starting with b6ba1b40ce4a88d4062bcab08d8b39f3078ca2e5b895898b7057c3a17bd90b00 not found: ID does not exist" Oct 30 01:23:33.883055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.883034 5938 scope.go:122] "RemoveContainer" containerID="e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db" Oct 30 01:23:33.883283 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:33.883263 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db\": container with ID starting with e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db not found: ID does not exist" containerID="e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db" Oct 30 01:23:33.883332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.883289 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db"} err="failed to get container status \"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db\": rpc error: code = NotFound desc = could not find container \"e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db\": container with ID starting with e294291f4b70ee2d8463d6dddd2ea734af6602aaa164806f588b6dae06f6b6db not found: ID does not exist" Oct 30 01:23:33.909460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.909436 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:23:33.914029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:33.914002 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f64dc0-openshift-default-59d7879cd9-f5nrs"] Oct 30 01:23:34.918524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:34.918491 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:35.351722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:35.351641 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:35.876185 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:35.876154 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" path="/var/lib/kubelet/pods/9452a925-9485-4073-bc8c-bf81a4d79275/volumes" Oct 30 01:23:37.095816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:37.095781 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:38.372082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.372048 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:38.510665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.510636 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:38.715976 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.715908 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:38.775091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.773054 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:23:38.775091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.773795 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" containerID="cri-o://60ab0c98576dd9241ed65f576cb14189c20153321198fa61ac849342278482d9" gracePeriod=30 Oct 30 01:23:38.802608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.802574 5938 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:38.802740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.802626 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:38.863022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:38.862995 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.265255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265222 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7"] Oct 30 01:23:39.265660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265647 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265662 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265673 5938 cpu_manager.go:414] "containerMap: removing container" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265678 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265684 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0fd93939-b863-4d83-91a7-03ed30bdf7a0" containerName="revoke-keys" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265690 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd93939-b863-4d83-91a7-03ed30bdf7a0" containerName="revoke-keys" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265706 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" Oct 30 01:23:39.265709 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265711 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" Oct 30 01:23:39.265958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265765 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265774 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="9452a925-9485-4073-bc8c-bf81a4d79275" containerName="istio-proxy" Oct 30 01:23:39.265958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265781 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd93939-b863-4d83-91a7-03ed30bdf7a0" containerName="revoke-keys" Oct 30 01:23:39.265958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265791 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6823d9-fa04-412f-940e-97018cb6d90a" containerName="payload-processing" Oct 30 01:23:39.265958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.265798 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e48bff3-a057-4901-bb9a-8efa404e8ae9" containerName="istio-proxy" Oct 30 01:23:39.270755 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.270734 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.288897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.285076 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:39.288897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.285130 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:39.288897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.287547 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7"] Oct 30 01:23:39.339514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.339463 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:39.339630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.339532 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:39.382978 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.382954 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.383270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.382998 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.383270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.383131 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxwd\" (UniqueName: \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.483551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.483527 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxwd\" (UniqueName: \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.483681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.483572 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.483775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.483694 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.484285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.484265 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.486048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.486029 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.506398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.506371 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxwd\" (UniqueName: \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.588286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.588237 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:39.712702 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.712675 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:23:39.741275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.741248 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:23:39.884133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.884028 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7"] Oct 30 01:23:39.887379 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:39.887217 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368531a1_8065_4335_a5c7_c2249626155b.slice/crio-2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265 WatchSource:0}: Error finding container 2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265: Status 404 returned error can't find the container with id 2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265 Oct 30 01:23:39.887631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.887602 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.905897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:39.905847 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" event={"ID":"368531a1-8065-4335-a5c7-c2249626155b","Type":"ContainerStarted","Data":"2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265"} Oct 30 01:23:40.025517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:40.025492 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.616990 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:40.616959 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.912449 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:40.912385 5938 generic.go:356] "Generic (PLEG): container finished" podID="368531a1-8065-4335-a5c7-c2249626155b" containerID="28a7cd1201ee7c4e276a6d9c606ec121f367747d14046088a8bb1e1de173c920" exitCode=0 Oct 30 01:23:40.912621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:40.912469 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" event={"ID":"368531a1-8065-4335-a5c7-c2249626155b","Type":"ContainerDied","Data":"28a7cd1201ee7c4e276a6d9c606ec121f367747d14046088a8bb1e1de173c920"} Oct 30 01:23:41.294594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:41.294537 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:41.630561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:41.630531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:42.055922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.055901 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:42.105715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.105688 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\") pod \"368531a1-8065-4335-a5c7-c2249626155b\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " Oct 30 01:23:42.105849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.105744 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\" (UniqueName: \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\") pod \"368531a1-8065-4335-a5c7-c2249626155b\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " Oct 30 01:23:42.105955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.105895 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\") pod \"368531a1-8065-4335-a5c7-c2249626155b\" (UID: \"368531a1-8065-4335-a5c7-c2249626155b\") " Oct 30 01:23:42.106076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.106042 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca" pod "368531a1-8065-4335-a5c7-c2249626155b" (UID: "368531a1-8065-4335-a5c7-c2249626155b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:42.106268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.106245 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/368531a1-8065-4335-a5c7-c2249626155b-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:42.107893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.107857 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca" pod "368531a1-8065-4335-a5c7-c2249626155b" (UID: "368531a1-8065-4335-a5c7-c2249626155b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:42.108002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.107953 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd" pod "368531a1-8065-4335-a5c7-c2249626155b" (UID: "368531a1-8065-4335-a5c7-c2249626155b"). InnerVolumeSpecName "kube-api-access-rsxwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:42.207155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.207097 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxwd\" (UniqueName: \"kubernetes.io/projected/368531a1-8065-4335-a5c7-c2249626155b-kube-api-access-rsxwd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:42.207155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.207128 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/368531a1-8065-4335-a5c7-c2249626155b-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:42.314441 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.314412 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:42.809930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.809901 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:42.902922 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.902895 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4"] Oct 30 01:23:42.914308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.914284 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a3619c-x9ws4"] Oct 30 01:23:42.923791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.923762 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" event={"ID":"368531a1-8065-4335-a5c7-c2249626155b","Type":"ContainerDied","Data":"2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265"} Oct 30 01:23:42.923950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.923817 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c8dc27a9871f0b5ba7359f476dfeb33ba97abc3b4fe63514757ee9c2595e265" Oct 30 01:23:42.923950 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.923771 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7" Oct 30 01:23:42.950299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:42.950275 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:43.459891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:43.459848 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:43.874490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:43.874462 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd93939-b863-4d83-91a7-03ed30bdf7a0" path="/var/lib/kubelet/pods/0fd93939-b863-4d83-91a7-03ed30bdf7a0/volumes" Oct 30 01:23:43.931800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:43.931771 5938 generic.go:356] "Generic (PLEG): container finished" podID="323a82af-703b-4887-8832-8187a65a33b2" containerID="60ab0c98576dd9241ed65f576cb14189c20153321198fa61ac849342278482d9" exitCode=0 Oct 30 01:23:43.931997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:43.931820 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerDied","Data":"60ab0c98576dd9241ed65f576cb14189c20153321198fa61ac849342278482d9"} Oct 30 01:23:43.931997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:43.931883 5938 scope.go:122] "RemoveContainer" containerID="2e9f31befbfca22182200433ac9489645db4fa75faf10be1ba5000350aa50799" Oct 30 01:23:44.033140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.033117 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:23:44.123763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123738 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.123905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123788 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.123905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123822 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.123905 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123856 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123922 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.123976 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124077 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124145 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\") pod \"323a82af-703b-4887-8832-8187a65a33b2\" (UID: \"323a82af-703b-4887-8832-8187a65a33b2\") " Oct 30 01:23:44.124269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124175 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:44.124269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124208 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:44.124269 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124255 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:44.124540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:44.124540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124503 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:44.124659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124566 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.124659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124584 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323a82af-703b-4887-8832-8187a65a33b2-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.124659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124599 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.124659 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.124613 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.126241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.126217 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:44.126618 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.126592 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:44.126751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.126687 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "kube-api-access-df6rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:44.126751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.126714 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token" pod "323a82af-703b-4887-8832-8187a65a33b2" (UID: "323a82af-703b-4887-8832-8187a65a33b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:44.225252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.225229 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6rh\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-kube-api-access-df6rh\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.225252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.225251 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323a82af-703b-4887-8832-8187a65a33b2-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.225389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.225261 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.225389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.225269 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323a82af-703b-4887-8832-8187a65a33b2-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.225389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.225278 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323a82af-703b-4887-8832-8187a65a33b2-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:44.713325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.713287 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" containerID="cri-o://eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e" gracePeriod=25 Oct 30 01:23:44.742008 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.741976 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" containerID="cri-o://7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af" gracePeriod=25 Oct 30 01:23:44.790681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.790654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:44.942334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.942312 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" Oct 30 01:23:44.942675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.942338 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w" event={"ID":"323a82af-703b-4887-8832-8187a65a33b2","Type":"ContainerDied","Data":"bddcc732059f9c3ffbbad446105b605881099c9727c092d6d8bb855832f47e93"} Oct 30 01:23:44.942675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:44.942380 5938 scope.go:122] "RemoveContainer" containerID="60ab0c98576dd9241ed65f576cb14189c20153321198fa61ac849342278482d9" Oct 30 01:23:45.003033 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:45.002969 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:23:45.014364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:45.014342 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-dgw5w"] Oct 30 01:23:45.254458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:45.254380 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:45.875144 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:45.875108 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323a82af-703b-4887-8832-8187a65a33b2" path="/var/lib/kubelet/pods/323a82af-703b-4887-8832-8187a65a33b2/volumes" Oct 30 01:23:46.269662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:46.269589 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:46.499769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:46.499740 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:47.238614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:47.238581 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:48.283131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:48.283103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.816520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:49.816478 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.960812 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:49.960776 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.114557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.114526 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.293317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.293285 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:23:50.293555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.293524 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" containerID="cri-o://712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6" gracePeriod=30 Oct 30 01:23:50.367186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.367103 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.512811 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.512778 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:50.844263 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:50.844183 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.002563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.002533 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.225109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.225080 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7"] Oct 30 01:23:51.229581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.229558 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.236548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.236527 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7f92f6-gkdz7"] Oct 30 01:23:51.291801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.291779 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx"] Oct 30 01:23:51.292249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292231 5938 cpu_manager.go:414] "containerMap: removing container" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:51.292319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292250 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:51.292319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292272 5938 cpu_manager.go:414] "containerMap: removing container" podUID="368531a1-8065-4335-a5c7-c2249626155b" containerName="revoke-keys" Oct 30 01:23:51.292319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292276 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368531a1-8065-4335-a5c7-c2249626155b" containerName="revoke-keys" Oct 30 01:23:51.292418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292354 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:51.292418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.292362 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="368531a1-8065-4335-a5c7-c2249626155b" containerName="revoke-keys" Oct 30 01:23:51.299219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.298617 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.313756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.313729 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx"] Oct 30 01:23:51.313756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.313747 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:51.314032 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.314014 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:51.314480 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.314458 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:51.323257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.323237 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:51.402602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.402575 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.402735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.402718 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.402792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.402769 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vnv\" (UniqueName: \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.503771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.503670 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.503771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.503730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vnv\" (UniqueName: \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.503771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.503756 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.504454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.504425 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.506042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.506018 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.526102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.526077 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vnv\" (UniqueName: \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\") pod \"maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.614251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.614226 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:51.732024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.731993 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.875217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.875191 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368531a1-8065-4335-a5c7-c2249626155b" path="/var/lib/kubelet/pods/368531a1-8065-4335-a5c7-c2249626155b/volumes" Oct 30 01:23:51.911282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.911159 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx"] Oct 30 01:23:51.913521 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:51.913497 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef75d164_1efe_451b_8443_993387f713a8.slice/crio-ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a WatchSource:0}: Error finding container ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a: Status 404 returned error can't find the container with id ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a Oct 30 01:23:51.983528 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:51.983500 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" event={"ID":"ef75d164-1efe-451b-8443-993387f713a8","Type":"ContainerStarted","Data":"ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a"} Oct 30 01:23:52.495141 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:52.495057 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:52.916046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:52.916017 5938 patch_prober.go:29] interesting pod/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:52.916458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:52.916064 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:52.990504 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:52.990474 5938 generic.go:356] "Generic (PLEG): container finished" podID="ef75d164-1efe-451b-8443-993387f713a8" containerID="0de65f77e4e5b48fd9a53df69194fa259eab7146836255a14fb618b1614efdf0" exitCode=0 Oct 30 01:23:52.990639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:52.990563 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" event={"ID":"ef75d164-1efe-451b-8443-993387f713a8","Type":"ContainerDied","Data":"0de65f77e4e5b48fd9a53df69194fa259eab7146836255a14fb618b1614efdf0"} Oct 30 01:23:54.129819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.129789 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.132599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.132577 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:54.228347 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.228323 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\") pod \"ef75d164-1efe-451b-8443-993387f713a8\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " Oct 30 01:23:54.228469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.228448 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\") pod \"ef75d164-1efe-451b-8443-993387f713a8\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " Oct 30 01:23:54.228510 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.228490 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\" (UniqueName: \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\") pod \"ef75d164-1efe-451b-8443-993387f713a8\" (UID: \"ef75d164-1efe-451b-8443-993387f713a8\") " Oct 30 01:23:54.228781 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.228760 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca" pod "ef75d164-1efe-451b-8443-993387f713a8" (UID: "ef75d164-1efe-451b-8443-993387f713a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:54.230313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.230291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca" pod "ef75d164-1efe-451b-8443-993387f713a8" (UID: "ef75d164-1efe-451b-8443-993387f713a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:54.230652 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.230631 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv" pod "ef75d164-1efe-451b-8443-993387f713a8" (UID: "ef75d164-1efe-451b-8443-993387f713a8"). InnerVolumeSpecName "kube-api-access-55vnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:54.329968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.329910 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vnv\" (UniqueName: \"kubernetes.io/projected/ef75d164-1efe-451b-8443-993387f713a8-kube-api-access-55vnv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:54.329968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.329935 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef75d164-1efe-451b-8443-993387f713a8-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:54.329968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:54.329947 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef75d164-1efe-451b-8443-993387f713a8-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.003049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.003012 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" event={"ID":"ef75d164-1efe-451b-8443-993387f713a8","Type":"ContainerDied","Data":"ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a"} Oct 30 01:23:55.003249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.003051 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx" Oct 30 01:23:55.003249 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.003062 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce612bc1fbf07bd3c6cba2e877d32a59a68da8a62746422242d15718d6d7064a" Oct 30 01:23:55.253964 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.253885 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:55.557429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.557403 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:23:55.642694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642671 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.642836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642708 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.642836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642745 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.642836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642787 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.642836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642830 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.643092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642884 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.643092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.642948 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.643092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643007 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.643092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643044 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\") pod \"0dd4836f-89b7-4245-8394-35dca2763ecd\" (UID: \"0dd4836f-89b7-4245-8394-35dca2763ecd\") " Oct 30 01:23:55.643092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643045 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:55.643368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643295 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:55.643368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643345 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:55.643486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643371 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:55.643486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643434 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.643598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643487 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.643598 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.643507 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:55.645211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.645191 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:55.645364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.645335 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "kube-api-access-xz5g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:55.645556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.645536 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:55.645649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.645581 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo" pod "0dd4836f-89b7-4245-8394-35dca2763ecd" (UID: "0dd4836f-89b7-4245-8394-35dca2763ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:23:55.743901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743878 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.743901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743899 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.744018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743909 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.744018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743918 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.744018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743926 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.744018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743934 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd4836f-89b7-4245-8394-35dca2763ecd-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.744018 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.743944 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz5g2\" (UniqueName: \"kubernetes.io/projected/0dd4836f-89b7-4245-8394-35dca2763ecd-kube-api-access-xz5g2\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:23:55.782170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.782149 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 01:23:55.793575 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.793521 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 01:23:55.889313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:55.889284 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.010306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.010282 5938 generic.go:356] "Generic (PLEG): container finished" podID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerID="712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6" exitCode=0 Oct 30 01:23:56.010419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.010330 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" Oct 30 01:23:56.010419 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.010369 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerDied","Data":"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6"} Oct 30 01:23:56.010499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.010422 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t" event={"ID":"0dd4836f-89b7-4245-8394-35dca2763ecd","Type":"ContainerDied","Data":"fc2906e145ffd1e1f8295e041ede2ed9e73a2c66e1a33adc15e6e891c4c18a75"} Oct 30 01:23:56.010499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.010444 5938 scope.go:122] "RemoveContainer" containerID="712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6" Oct 30 01:23:56.020176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.020157 5938 scope.go:122] "RemoveContainer" containerID="c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b" Oct 30 01:23:56.028574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.028557 5938 scope.go:122] "RemoveContainer" containerID="712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6" Oct 30 01:23:56.028821 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:56.028798 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6\": container with ID starting with 712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6 not found: ID does not exist" containerID="712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6" Oct 30 01:23:56.028884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.028829 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6"} err="failed to get container status \"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6\": rpc error: code = NotFound desc = could not find container \"712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6\": container with ID starting with 712b8c4d9b7838934115aca39bf694c019ea9e3a2ddf95f89c7eca6d81b38ca6 not found: ID does not exist" Oct 30 01:23:56.028884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.028846 5938 scope.go:122] "RemoveContainer" containerID="c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b" Oct 30 01:23:56.029092 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:56.029075 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b\": container with ID starting with c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b not found: ID does not exist" containerID="c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b" Oct 30 01:23:56.029138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.029099 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b"} err="failed to get container status \"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b\": rpc error: code = NotFound desc = could not find container \"c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b\": container with ID starting with c24eb0da763b291ae4b7e3eb2f44bdc5f4999e85942e9cefba2fd9e32be9b50b not found: ID does not exist" Oct 30 01:23:56.060155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.060105 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:23:56.063822 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.063798 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a3619c-openshift-default-675f567c95-7ts8t"] Oct 30 01:23:56.064239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.064219 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.269344 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:56.269319 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:23:57.595113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:57.595071 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:57.874301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:57.874267 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" path="/var/lib/kubelet/pods/0dd4836f-89b7-4245-8394-35dca2763ecd/volumes" Oct 30 01:23:58.016110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.016077 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.156558 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.156497 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.354472 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.354425 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.807608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.807580 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.832680 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:23:58.832649 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:23:58.914330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914306 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4"] Oct 30 01:23:58.914707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914691 5938 cpu_manager.go:414] "containerMap: removing container" podUID="ef75d164-1efe-451b-8443-993387f713a8" containerName="revoke-keys" Oct 30 01:23:58.914773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914710 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef75d164-1efe-451b-8443-993387f713a8" containerName="revoke-keys" Oct 30 01:23:58.914773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914751 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:23:58.914773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914760 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:23:58.914917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914777 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:23:58.914917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914786 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:23:58.914917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914801 5938 cpu_manager.go:414] "containerMap: removing container" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:58.914917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914809 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:58.915052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914941 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:23:58.915052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914955 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="323a82af-703b-4887-8832-8187a65a33b2" containerName="istio-proxy" Oct 30 01:23:58.915052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.914964 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef75d164-1efe-451b-8443-993387f713a8" containerName="revoke-keys" Oct 30 01:23:58.918091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.918074 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:58.949322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.949291 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:58.949433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.949321 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:58.951087 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:58.951059 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4"] Oct 30 01:23:59.006192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.006168 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:23:59.006313 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.006294 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:23:59.072437 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.072381 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.072556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.072495 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcwc\" (UniqueName: \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.072556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.072543 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.173573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.173548 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slcwc\" (UniqueName: \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.173588 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.173688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.173617 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.174334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.174313 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.175938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.175913 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.195564 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.195544 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcwc\" (UniqueName: \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\") pod \"maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.231952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.231930 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:23:59.444071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.444034 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:59.528186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:23:59.528165 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4"] Oct 30 01:23:59.531063 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:23:59.531029 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc783d296_f1af_4b12_8cfa_30865f511375.slice/crio-d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1 WatchSource:0}: Error finding container d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1: Status 404 returned error can't find the container with id d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1 Oct 30 01:24:00.035044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:00.034967 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" event={"ID":"c783d296-f1af-4b12-8cfa-30865f511375","Type":"ContainerStarted","Data":"c252cc5370e761be20e505bcfbffa54a908896908cc6d6d0f50b53f0c23671cd"} Oct 30 01:24:00.035044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:00.035028 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" event={"ID":"c783d296-f1af-4b12-8cfa-30865f511375","Type":"ContainerStarted","Data":"d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1"} Oct 30 01:24:00.074759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:00.074714 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" podStartSLOduration=2.074699964 podStartE2EDuration="2.074699964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:00.073092329 +0000 UTC m=+3092.313327443" watchObservedRunningTime="2026-10-30 01:24:00.074699964 +0000 UTC m=+3092.314935099" Oct 30 01:24:01.043103 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:01.043060 5938 generic.go:356] "Generic (PLEG): container finished" podID="c783d296-f1af-4b12-8cfa-30865f511375" containerID="c252cc5370e761be20e505bcfbffa54a908896908cc6d6d0f50b53f0c23671cd" exitCode=0 Oct 30 01:24:01.043537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:01.043148 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" event={"ID":"c783d296-f1af-4b12-8cfa-30865f511375","Type":"ContainerDied","Data":"c252cc5370e761be20e505bcfbffa54a908896908cc6d6d0f50b53f0c23671cd"} Oct 30 01:24:01.239160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:01.239125 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:01.695455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:01.695418 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 01:24:01.871126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:01.871096 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:02.017081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.017005 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:02.178024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.177999 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:24:02.299999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.299939 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\") pod \"c783d296-f1af-4b12-8cfa-30865f511375\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " Oct 30 01:24:02.300131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.300025 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\") pod \"c783d296-f1af-4b12-8cfa-30865f511375\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " Oct 30 01:24:02.300131 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.300106 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\" (UniqueName: \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\") pod \"c783d296-f1af-4b12-8cfa-30865f511375\" (UID: \"c783d296-f1af-4b12-8cfa-30865f511375\") " Oct 30 01:24:02.300359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.300334 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca" pod "c783d296-f1af-4b12-8cfa-30865f511375" (UID: "c783d296-f1af-4b12-8cfa-30865f511375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:02.302088 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.302060 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca" pod "c783d296-f1af-4b12-8cfa-30865f511375" (UID: "c783d296-f1af-4b12-8cfa-30865f511375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:02.302191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.302115 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc" pod "c783d296-f1af-4b12-8cfa-30865f511375" (UID: "c783d296-f1af-4b12-8cfa-30865f511375"). InnerVolumeSpecName "kube-api-access-slcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:02.401171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.401146 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:02.401171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.401167 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slcwc\" (UniqueName: \"kubernetes.io/projected/c783d296-f1af-4b12-8cfa-30865f511375-kube-api-access-slcwc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:02.401299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.401177 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c783d296-f1af-4b12-8cfa-30865f511375-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:02.617438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.617407 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm"] Oct 30 01:24:02.617832 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.617818 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c783d296-f1af-4b12-8cfa-30865f511375" containerName="revoke-keys" Oct 30 01:24:02.617909 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.617833 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c783d296-f1af-4b12-8cfa-30865f511375" containerName="revoke-keys" Oct 30 01:24:02.617952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.617924 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c783d296-f1af-4b12-8cfa-30865f511375" containerName="revoke-keys" Oct 30 01:24:02.617952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.617937 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd4836f-89b7-4245-8394-35dca2763ecd" containerName="istio-proxy" Oct 30 01:24:02.620917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.620903 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.638971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.638946 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm"] Oct 30 01:24:02.703823 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.703796 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8xl\" (UniqueName: \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.703980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.703932 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.704040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.703984 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.805191 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.805165 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.805311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.805234 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8xl\" (UniqueName: \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.805311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.805304 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.805887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.805841 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.807555 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.807533 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.829329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.829303 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8xl\" (UniqueName: \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\") pod \"maas-api-revoke-sub-0d76e889-xc5qm\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.933959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.933910 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:02.986958 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:02.986932 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:03.054906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.054857 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" Oct 30 01:24:03.054906 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.054891 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4" event={"ID":"c783d296-f1af-4b12-8cfa-30865f511375","Type":"ContainerDied","Data":"d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1"} Oct 30 01:24:03.055067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.054940 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d672a998fed5346ee0ff17a174bbcf39f429337263bf90a2eb39737f2089efb1" Oct 30 01:24:03.227369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.227346 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm"] Oct 30 01:24:03.229994 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:24:03.229970 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a60250c_249e_4105_8828_d5b3e1d77628.slice/crio-3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa WatchSource:0}: Error finding container 3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa: Status 404 returned error can't find the container with id 3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa Oct 30 01:24:03.724564 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.724537 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 01:24:03.732899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.732856 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 01:24:03.885858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:03.885793 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:04.061345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:04.061315 5938 generic.go:356] "Generic (PLEG): container finished" podID="3a60250c-249e-4105-8828-d5b3e1d77628" containerID="d3564ac978b4fa35520daec865648b64c6ebfe5bb75be77c74a021ccfd98f923" exitCode=0 Oct 30 01:24:04.061490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:04.061402 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" event={"ID":"3a60250c-249e-4105-8828-d5b3e1d77628","Type":"ContainerDied","Data":"d3564ac978b4fa35520daec865648b64c6ebfe5bb75be77c74a021ccfd98f923"} Oct 30 01:24:04.061490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:04.061446 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" event={"ID":"3a60250c-249e-4105-8828-d5b3e1d77628","Type":"ContainerStarted","Data":"3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa"} Oct 30 01:24:04.780532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:04.780499 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:04.989969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:04.989944 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:24:05.014025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.014005 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:24:05.069070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.069043 5938 generic.go:356] "Generic (PLEG): container finished" podID="655a8378-d816-40a9-832b-25b1b6b211a9" containerID="eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e" exitCode=0 Oct 30 01:24:05.069230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.069090 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" Oct 30 01:24:05.069230 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.069156 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" event={"ID":"655a8378-d816-40a9-832b-25b1b6b211a9","Type":"ContainerDied","Data":"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e"} Oct 30 01:24:05.069352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.069237 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw" event={"ID":"655a8378-d816-40a9-832b-25b1b6b211a9","Type":"ContainerDied","Data":"12ef92ffe2dcd3cc9ceb8f86007c03d139df3dfaf1c7ceffd50e6b5690c8ebec"} Oct 30 01:24:05.069352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.069297 5938 scope.go:122] "RemoveContainer" containerID="eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e" Oct 30 01:24:05.071908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.071544 5938 generic.go:356] "Generic (PLEG): container finished" podID="4236541a-c30a-4c70-8220-57f280c7389d" containerID="7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af" exitCode=0 Oct 30 01:24:05.071908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.071607 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" Oct 30 01:24:05.071908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.071655 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" event={"ID":"4236541a-c30a-4c70-8220-57f280c7389d","Type":"ContainerDied","Data":"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af"} Oct 30 01:24:05.071908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.071689 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2" event={"ID":"4236541a-c30a-4c70-8220-57f280c7389d","Type":"ContainerDied","Data":"384152c2a570930a93af780ce52a5b28ca99d20a1bede5f62361f63c0322d333"} Oct 30 01:24:05.080450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.080427 5938 scope.go:122] "RemoveContainer" containerID="eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e" Oct 30 01:24:05.080796 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:05.080773 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e\": container with ID starting with eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e not found: ID does not exist" containerID="eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e" Oct 30 01:24:05.080856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.080809 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e"} err="failed to get container status \"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e\": rpc error: code = NotFound desc = could not find container \"eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e\": container with ID starting with eae65a91a323628a0c74c0165d2c7ded9fe60e88be18233bd3bddf10feb40c2e not found: ID does not exist" Oct 30 01:24:05.080856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.080833 5938 scope.go:122] "RemoveContainer" containerID="7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af" Oct 30 01:24:05.091750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.091722 5938 scope.go:122] "RemoveContainer" containerID="7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af" Oct 30 01:24:05.092069 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:05.092050 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af\": container with ID starting with 7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af not found: ID does not exist" containerID="7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af" Oct 30 01:24:05.092166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.092081 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af"} err="failed to get container status \"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af\": rpc error: code = NotFound desc = could not find container \"7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af\": container with ID starting with 7a3a72a94e0190199f4d5f0bac3a9b3e03d0364e1afd24658d0f8e25ba4e98af not found: ID does not exist" Oct 30 01:24:05.126174 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126150 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\" (UniqueName: \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\") pod \"655a8378-d816-40a9-832b-25b1b6b211a9\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " Oct 30 01:24:05.126316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126247 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\" (UniqueName: \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\") pod \"4236541a-c30a-4c70-8220-57f280c7389d\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " Oct 30 01:24:05.126316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126308 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\") pod \"655a8378-d816-40a9-832b-25b1b6b211a9\" (UID: \"655a8378-d816-40a9-832b-25b1b6b211a9\") " Oct 30 01:24:05.126446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126350 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\") pod \"4236541a-c30a-4c70-8220-57f280c7389d\" (UID: \"4236541a-c30a-4c70-8220-57f280c7389d\") " Oct 30 01:24:05.127007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126916 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume" pod "655a8378-d816-40a9-832b-25b1b6b211a9" (UID: "655a8378-d816-40a9-832b-25b1b6b211a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:05.127007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.126970 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume" pod "4236541a-c30a-4c70-8220-57f280c7389d" (UID: "4236541a-c30a-4c70-8220-57f280c7389d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:05.128733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.128705 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc" pod "655a8378-d816-40a9-832b-25b1b6b211a9" (UID: "655a8378-d816-40a9-832b-25b1b6b211a9"). InnerVolumeSpecName "kube-api-access-jvvkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:05.130624 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.130602 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj" pod "4236541a-c30a-4c70-8220-57f280c7389d" (UID: "4236541a-c30a-4c70-8220-57f280c7389d"). InnerVolumeSpecName "kube-api-access-v7pcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:05.194942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.194919 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:05.227826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.227778 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvvkc\" (UniqueName: \"kubernetes.io/projected/655a8378-d816-40a9-832b-25b1b6b211a9-kube-api-access-jvvkc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.227826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.227799 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7pcj\" (UniqueName: \"kubernetes.io/projected/4236541a-c30a-4c70-8220-57f280c7389d-kube-api-access-v7pcj\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.227826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.227811 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655a8378-d816-40a9-832b-25b1b6b211a9-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.227826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.227820 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4236541a-c30a-4c70-8220-57f280c7389d-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.328733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.328711 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\") pod \"3a60250c-249e-4105-8828-d5b3e1d77628\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " Oct 30 01:24:05.328897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.328775 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\" (UniqueName: \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\") pod \"3a60250c-249e-4105-8828-d5b3e1d77628\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " Oct 30 01:24:05.328897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.328854 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\") pod \"3a60250c-249e-4105-8828-d5b3e1d77628\" (UID: \"3a60250c-249e-4105-8828-d5b3e1d77628\") " Oct 30 01:24:05.329207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.329187 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca" pod "3a60250c-249e-4105-8828-d5b3e1d77628" (UID: "3a60250c-249e-4105-8828-d5b3e1d77628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:05.330725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.330706 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca" pod "3a60250c-249e-4105-8828-d5b3e1d77628" (UID: "3a60250c-249e-4105-8828-d5b3e1d77628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:05.330788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.330724 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl" pod "3a60250c-249e-4105-8828-d5b3e1d77628" (UID: "3a60250c-249e-4105-8828-d5b3e1d77628"). InnerVolumeSpecName "kube-api-access-st8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:05.415391 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.415365 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:24:05.418894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.418847 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a3619c-746d6d556d-rxcbw"] Oct 30 01:24:05.429705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.429686 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.429788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.429709 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st8xl\" (UniqueName: \"kubernetes.io/projected/3a60250c-249e-4105-8828-d5b3e1d77628-kube-api-access-st8xl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.429788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.429719 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a60250c-249e-4105-8828-d5b3e1d77628-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:05.451738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.451717 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:24:05.454935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.454912 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a3619c-6584cd5f87-cgfd2"] Oct 30 01:24:05.874838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.874804 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4236541a-c30a-4c70-8220-57f280c7389d" path="/var/lib/kubelet/pods/4236541a-c30a-4c70-8220-57f280c7389d/volumes" Oct 30 01:24:05.875218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:05.875202 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" path="/var/lib/kubelet/pods/655a8378-d816-40a9-832b-25b1b6b211a9/volumes" Oct 30 01:24:06.079283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:06.079218 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" Oct 30 01:24:06.079283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:06.079253 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm" event={"ID":"3a60250c-249e-4105-8828-d5b3e1d77628","Type":"ContainerDied","Data":"3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa"} Oct 30 01:24:06.079468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:06.079311 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f30fdf1ec19a8a49523780e903ab37d029406f15adff6d04cce328e6d4550fa" Oct 30 01:24:06.474770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:06.474736 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:06.696737 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:06.696684 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" containerID="cri-o://a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157" gracePeriod=25 Oct 30 01:24:08.296067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:08.296036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:09.313400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:09.313372 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 01:24:09.347063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:09.347038 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 01:24:09.502356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:09.502317 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:09.651623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:09.651593 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:09.978669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:09.978584 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:10.801940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:10.801899 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" containerID="cri-o://2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2" gracePeriod=45 Oct 30 01:24:10.817285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:10.817244 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" containerID="cri-o://be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb" gracePeriod=45 Oct 30 01:24:11.059506 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.059453 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 01:24:11.090540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.090518 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 01:24:11.113051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.113019 5938 generic.go:356] "Generic (PLEG): container finished" podID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerID="2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2" exitCode=0 Oct 30 01:24:11.113210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.113077 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" Oct 30 01:24:11.113210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.113108 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" event={"ID":"6f8f9c58-1e95-40cd-911f-7fdaa5052c58","Type":"ContainerDied","Data":"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2"} Oct 30 01:24:11.113210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.113149 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc" event={"ID":"6f8f9c58-1e95-40cd-911f-7fdaa5052c58","Type":"ContainerDied","Data":"6003bf2b11109c34df96a747078e20962232051367e97c6c20106aba5b35bdb6"} Oct 30 01:24:11.113210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.113168 5938 scope.go:122] "RemoveContainer" containerID="2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2" Oct 30 01:24:11.114792 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.114770 5938 generic.go:356] "Generic (PLEG): container finished" podID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerID="be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb" exitCode=0 Oct 30 01:24:11.114924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.114811 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" event={"ID":"492caa93-1750-4f0a-85ad-1bcea6a4aea0","Type":"ContainerDied","Data":"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb"} Oct 30 01:24:11.114924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.114826 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" Oct 30 01:24:11.114924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.114835 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh" event={"ID":"492caa93-1750-4f0a-85ad-1bcea6a4aea0","Type":"ContainerDied","Data":"463894360d8ec7196804045ca86fa6b87f0cf148bbce750e0b9209a49e93b421"} Oct 30 01:24:11.122373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.122350 5938 scope.go:122] "RemoveContainer" containerID="2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2" Oct 30 01:24:11.122680 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:11.122648 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2\": container with ID starting with 2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2 not found: ID does not exist" containerID="2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2" Oct 30 01:24:11.122791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.122689 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2"} err="failed to get container status \"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2\": rpc error: code = NotFound desc = could not find container \"2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2\": container with ID starting with 2f11df4d7b9c4cf70a132af3cac1b5d2903c253e7ac5ec2e8452f4088b7daeb2 not found: ID does not exist" Oct 30 01:24:11.122791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.122714 5938 scope.go:122] "RemoveContainer" containerID="be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb" Oct 30 01:24:11.133237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.133212 5938 scope.go:122] "RemoveContainer" containerID="be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb" Oct 30 01:24:11.133597 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:11.133574 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb\": container with ID starting with be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb not found: ID does not exist" containerID="be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb" Oct 30 01:24:11.133685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.133606 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb"} err="failed to get container status \"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb\": rpc error: code = NotFound desc = could not find container \"be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb\": container with ID starting with be1fe3cf5575eed84dd34985345ee7c94f9894efed05ba2c84ef4425940d15cb not found: ID does not exist" Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176379 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176532 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\" (UniqueName: \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176565 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176612 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176653 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.176679 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\") pod \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\" (UID: \"6f8f9c58-1e95-40cd-911f-7fdaa5052c58\") " Oct 30 01:24:11.177673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.177283 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.177673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.177381 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.177673 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.177399 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.179767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.179743 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:11.179911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.179840 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.180078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.180047 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl" pod "6f8f9c58-1e95-40cd-911f-7fdaa5052c58" (UID: "6f8f9c58-1e95-40cd-911f-7fdaa5052c58"). InnerVolumeSpecName "kube-api-access-t7msl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.278060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278033 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278228 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278089 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\" (UniqueName: \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278291 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.278305 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278298 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\" (UniqueName: \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278330 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278355 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278399 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\") pod \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\" (UID: \"492caa93-1750-4f0a-85ad-1bcea6a4aea0\") " Oct 30 01:24:11.278696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278652 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278701 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278722 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278733 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278744 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278755 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278766 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.278884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278779 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7msl\" (UniqueName: \"kubernetes.io/projected/6f8f9c58-1e95-40cd-911f-7fdaa5052c58-kube-api-access-t7msl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.279195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.278907 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.280319 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.280299 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:11.280612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.280595 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "kube-api-access-wq29j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.280671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.280624 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm" pod "492caa93-1750-4f0a-85ad-1bcea6a4aea0" (UID: "492caa93-1750-4f0a-85ad-1bcea6a4aea0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.379307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.379280 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492caa93-1750-4f0a-85ad-1bcea6a4aea0-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.379307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.379304 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.379456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.379316 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.379456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.379329 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492caa93-1750-4f0a-85ad-1bcea6a4aea0-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.379456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.379341 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq29j\" (UniqueName: \"kubernetes.io/projected/492caa93-1750-4f0a-85ad-1bcea6a4aea0-kube-api-access-wq29j\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.457199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.457174 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 01:24:11.469117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.469090 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/premium-w2-1a11cf-kserve-6954dd7bcf-slhqc"] Oct 30 01:24:11.470757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.470736 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn"] Oct 30 01:24:11.471184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471168 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471185 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471199 5938 cpu_manager.go:414] "containerMap: removing container" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471205 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471215 5938 cpu_manager.go:414] "containerMap: removing container" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471220 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471236 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3a60250c-249e-4105-8828-d5b3e1d77628" containerName="revoke-keys" Oct 30 01:24:11.471255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471241 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a60250c-249e-4105-8828-d5b3e1d77628" containerName="revoke-keys" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471261 5938 cpu_manager.go:414] "containerMap: removing container" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471266 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471318 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="655a8378-d816-40a9-832b-25b1b6b211a9" containerName="payload-processing" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471326 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4236541a-c30a-4c70-8220-57f280c7389d" containerName="payload-pre-processing" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471334 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" containerName="main" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471339 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" containerName="main" Oct 30 01:24:11.471490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.471346 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a60250c-249e-4105-8828-d5b3e1d77628" containerName="revoke-keys" Oct 30 01:24:11.475736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.475721 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.493159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.492424 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn"] Oct 30 01:24:11.493159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.492453 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:24:11.493159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.492459 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:24:11.493159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.492517 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:24:11.493159 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.492557 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:24:11.500533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.500511 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 01:24:11.505544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.505510 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1a11cf/simulator-w2-1a11cf-kserve-7f895d686-j8dnh"] Oct 30 01:24:11.580881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.580847 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.581022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.580898 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.581022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.580993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ps4d\" (UniqueName: \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.681686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.681621 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ps4d\" (UniqueName: \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.681795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.681703 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.681795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.681730 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.682302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.682278 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.684003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.683986 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.704067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.704043 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ps4d\" (UniqueName: \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\") pod \"maas-api-revoke-sub-a7ae2d26-jm2xn\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.788013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.787989 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:11.876058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.876033 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492caa93-1750-4f0a-85ad-1bcea6a4aea0" path="/var/lib/kubelet/pods/492caa93-1750-4f0a-85ad-1bcea6a4aea0/volumes" Oct 30 01:24:11.876458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:11.876443 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8f9c58-1e95-40cd-911f-7fdaa5052c58" path="/var/lib/kubelet/pods/6f8f9c58-1e95-40cd-911f-7fdaa5052c58/volumes" Oct 30 01:24:12.083977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:12.083915 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn"] Oct 30 01:24:12.087428 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:24:12.087400 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2de4cb_0fd2_49f5_b59c_1e5f303cb70f.slice/crio-f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272 WatchSource:0}: Error finding container f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272: Status 404 returned error can't find the container with id f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272 Oct 30 01:24:12.121732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:12.121705 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" event={"ID":"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f","Type":"ContainerStarted","Data":"f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272"} Oct 30 01:24:12.717468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:12.717402 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:13.130966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:13.130940 5938 generic.go:356] "Generic (PLEG): container finished" podID="fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" containerID="2021c94a81954e77956d868663a8019952ad1346c04cc16ff8bfcc52ac6f1e09" exitCode=0 Oct 30 01:24:13.131306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:13.131035 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" event={"ID":"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f","Type":"ContainerDied","Data":"2021c94a81954e77956d868663a8019952ad1346c04cc16ff8bfcc52ac6f1e09"} Oct 30 01:24:14.264975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.264952 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:14.314616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.314585 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" containerID="cri-o://686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4" gracePeriod=25 Oct 30 01:24:14.347881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.347833 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" containerID="cri-o://998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1" gracePeriod=25 Oct 30 01:24:14.406641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.406582 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\" (UniqueName: \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\") pod \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " Oct 30 01:24:14.406641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.406635 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\") pod \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " Oct 30 01:24:14.406824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.406690 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\") pod \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\" (UID: \"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f\") " Oct 30 01:24:14.407060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.407032 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca" pod "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" (UID: "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:14.408837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.408813 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d" pod "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" (UID: "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f"). InnerVolumeSpecName "kube-api-access-7ps4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:14.409042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.409026 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca" pod "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" (UID: "fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:14.507301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.507282 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.507301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.507301 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.507445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:14.507311 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ps4d\" (UniqueName: \"kubernetes.io/projected/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f-kube-api-access-7ps4d\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:15.143321 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:15.143295 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" Oct 30 01:24:15.143476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:15.143316 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn" event={"ID":"fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f","Type":"ContainerDied","Data":"f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272"} Oct 30 01:24:15.143476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:15.143396 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f52b1a017b4acaf66e277ce0bb92458125b733234d48708588d9488060439272" Oct 30 01:24:16.519431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:16.519392 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.015335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.015301 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.107283 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.107250 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx"] Oct 30 01:24:17.118571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.118546 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-1a11cf-hbzwx"] Oct 30 01:24:17.153002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.152980 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.673696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.673666 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:17.874438 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:17.874410 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef75d164-1efe-451b-8443-993387f713a8" path="/var/lib/kubelet/pods/ef75d164-1efe-451b-8443-993387f713a8/volumes" Oct 30 01:24:18.744542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:18.744500 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" containerID="cri-o://a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e" gracePeriod=45 Oct 30 01:24:18.751787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:18.751756 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" containerID="cri-o://2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e" gracePeriod=45 Oct 30 01:24:18.998329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:18.998275 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 01:24:19.011718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.011692 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 01:24:19.040665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040636 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.040801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040684 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\" (UniqueName: \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.040801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.040801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040747 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.041001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040801 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.041001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040831 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.041001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040896 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\" (UniqueName: \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.041001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.040958 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\") pod \"7c8a428a-e570-46b0-a82d-838d0c76cad0\" (UID: \"7c8a428a-e570-46b0-a82d-838d0c76cad0\") " Oct 30 01:24:19.041227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041004 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.041227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041028 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.041227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041085 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.041227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041130 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\" (UniqueName: \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\") pod \"8094870f-eac3-4eae-922e-95a27b456eba\" (UID: \"8094870f-eac3-4eae-922e-95a27b456eba\") " Oct 30 01:24:19.042124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041571 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.042124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.041792 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.042124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.042040 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.042339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.042239 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.042769 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.042417 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.043058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.043034 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.044063 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.044037 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:19.044187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.044160 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:19.044630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.044607 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "kube-api-access-xksl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:19.044803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.044781 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm" pod "7c8a428a-e570-46b0-a82d-838d0c76cad0" (UID: "7c8a428a-e570-46b0-a82d-838d0c76cad0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.045100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.045079 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:19.045172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.045103 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9" pod "8094870f-eac3-4eae-922e-95a27b456eba" (UID: "8094870f-eac3-4eae-922e-95a27b456eba"). InnerVolumeSpecName "kube-api-access-jg8z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142599 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142638 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8094870f-eac3-4eae-922e-95a27b456eba-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142646 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142655 5938 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c8a428a-e570-46b0-a82d-838d0c76cad0-tls-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142664 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.142653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142671 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142680 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xksl9\" (UniqueName: \"kubernetes.io/projected/7c8a428a-e570-46b0-a82d-838d0c76cad0-kube-api-access-xksl9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142689 5938 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c8a428a-e570-46b0-a82d-838d0c76cad0-dshm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142697 5938 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-tmp-dir\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142704 5938 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-home\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142711 5938 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8094870f-eac3-4eae-922e-95a27b456eba-model-cache\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.143182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.142718 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8z9\" (UniqueName: \"kubernetes.io/projected/8094870f-eac3-4eae-922e-95a27b456eba-kube-api-access-jg8z9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:19.165942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.165914 5938 generic.go:356] "Generic (PLEG): container finished" podID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerID="2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e" exitCode=0 Oct 30 01:24:19.166068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.165965 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" Oct 30 01:24:19.166068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.165984 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" event={"ID":"7c8a428a-e570-46b0-a82d-838d0c76cad0","Type":"ContainerDied","Data":"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e"} Oct 30 01:24:19.166068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.166020 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s" event={"ID":"7c8a428a-e570-46b0-a82d-838d0c76cad0","Type":"ContainerDied","Data":"f72cb4fed4332d932ae250819652240be24335f4bc96b72e966c699ad3876a37"} Oct 30 01:24:19.166068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.166036 5938 scope.go:122] "RemoveContainer" containerID="2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e" Oct 30 01:24:19.167534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.167497 5938 generic.go:356] "Generic (PLEG): container finished" podID="8094870f-eac3-4eae-922e-95a27b456eba" containerID="a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e" exitCode=0 Oct 30 01:24:19.167640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.167527 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" event={"ID":"8094870f-eac3-4eae-922e-95a27b456eba","Type":"ContainerDied","Data":"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e"} Oct 30 01:24:19.167640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.167549 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" Oct 30 01:24:19.167640 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.167569 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79" event={"ID":"8094870f-eac3-4eae-922e-95a27b456eba","Type":"ContainerDied","Data":"0bf1b514797f31275f8bbf80c8fca3c57161c3eb684b85728d9abc8e21ff2671"} Oct 30 01:24:19.175043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.175019 5938 scope.go:122] "RemoveContainer" containerID="2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e" Oct 30 01:24:19.175303 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:19.175282 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e\": container with ID starting with 2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e not found: ID does not exist" containerID="2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e" Oct 30 01:24:19.175372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.175316 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e"} err="failed to get container status \"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e\": rpc error: code = NotFound desc = could not find container \"2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e\": container with ID starting with 2d5b8da164da9a2b8de9e23c2676e88fe75dbf8c77630b01edde5cb011a1c36e not found: ID does not exist" Oct 30 01:24:19.175372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.175339 5938 scope.go:122] "RemoveContainer" containerID="a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e" Oct 30 01:24:19.183760 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.183745 5938 scope.go:122] "RemoveContainer" containerID="a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e" Oct 30 01:24:19.184135 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:19.184116 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e\": container with ID starting with a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e not found: ID does not exist" containerID="a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e" Oct 30 01:24:19.184221 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.184140 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e"} err="failed to get container status \"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e\": rpc error: code = NotFound desc = could not find container \"a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e\": container with ID starting with a68d46037677845b44440717dd295b3005bb8f564a7889887a8b8b5f0d45228e not found: ID does not exist" Oct 30 01:24:19.214900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.214858 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 01:24:19.217542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.217519 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-w6-021d6d-kserve-d97574bd6-cbb79"] Oct 30 01:24:19.253001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.252946 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 01:24:19.256936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.256916 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/premium-w6-021d6d-kserve-69474b4b9f-qmb9s"] Oct 30 01:24:19.874913 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.874882 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" path="/var/lib/kubelet/pods/7c8a428a-e570-46b0-a82d-838d0c76cad0/volumes" Oct 30 01:24:19.875321 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:19.875305 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8094870f-eac3-4eae-922e-95a27b456eba" path="/var/lib/kubelet/pods/8094870f-eac3-4eae-922e-95a27b456eba/volumes" Oct 30 01:24:21.702856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:21.702815 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:21.710695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:21.710670 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:22.717652 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:22.717607 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:23.544569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:23.544510 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:23.727796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:23.727745 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:23.958705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:23.958675 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:24.677624 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:24.677585 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:25.182722 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.182689 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:25.276550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.276520 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4"] Oct 30 01:24:25.287972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.287944 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-021d6d-w4bx4"] Oct 30 01:24:25.323102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.323079 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:25.874434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.874395 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c783d296-f1af-4b12-8cfa-30865f511375" path="/var/lib/kubelet/pods/c783d296-f1af-4b12-8cfa-30865f511375/volumes" Oct 30 01:24:25.893324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:25.893295 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:26.951642 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:26.951618 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 01:24:27.011545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.011486 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\") pod \"fe67c1fd-69f6-41a5-a599-277101dfd095\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " Oct 30 01:24:27.011545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.011530 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\" (UniqueName: \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\") pod \"fe67c1fd-69f6-41a5-a599-277101dfd095\" (UID: \"fe67c1fd-69f6-41a5-a599-277101dfd095\") " Oct 30 01:24:27.011841 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.011819 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume" pod "fe67c1fd-69f6-41a5-a599-277101dfd095" (UID: "fe67c1fd-69f6-41a5-a599-277101dfd095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:27.013603 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.013575 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75" pod "fe67c1fd-69f6-41a5-a599-277101dfd095" (UID: "fe67c1fd-69f6-41a5-a599-277101dfd095"). InnerVolumeSpecName "kube-api-access-rjn75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:27.112683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.112656 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe67c1fd-69f6-41a5-a599-277101dfd095-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:27.112683 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.112681 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjn75\" (UniqueName: \"kubernetes.io/projected/fe67c1fd-69f6-41a5-a599-277101dfd095-kube-api-access-rjn75\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:27.216052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.216022 5938 generic.go:356] "Generic (PLEG): container finished" podID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerID="a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157" exitCode=0 Oct 30 01:24:27.216182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.216072 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" Oct 30 01:24:27.216182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.216110 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" event={"ID":"fe67c1fd-69f6-41a5-a599-277101dfd095","Type":"ContainerDied","Data":"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157"} Oct 30 01:24:27.216182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.216160 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm" event={"ID":"fe67c1fd-69f6-41a5-a599-277101dfd095","Type":"ContainerDied","Data":"1a6fa75823295e87f3b26acd13d9f7e2a1362210b811d88aeea3e3eedfe8fdde"} Oct 30 01:24:27.216182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.216182 5938 scope.go:122] "RemoveContainer" containerID="a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157" Oct 30 01:24:27.225984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.225967 5938 scope.go:122] "RemoveContainer" containerID="a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157" Oct 30 01:24:27.226236 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:27.226219 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157\": container with ID starting with a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157 not found: ID does not exist" containerID="a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157" Oct 30 01:24:27.226295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.226243 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157"} err="failed to get container status \"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157\": rpc error: code = NotFound desc = could not find container \"a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157\": container with ID starting with a7c27fb96d7f45abc20c893f48595e2635b8f089c8ecf065c52448dbb6585157 not found: ID does not exist" Oct 30 01:24:27.264828 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.264768 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 01:24:27.269592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.269559 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1a11cf-6d77fdc4b4-tqjbm"] Oct 30 01:24:27.874771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:27.874741 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" path="/var/lib/kubelet/pods/fe67c1fd-69f6-41a5-a599-277101dfd095/volumes" Oct 30 01:24:28.833276 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:28.833242 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:24:31.703229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:31.703183 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:31.710591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:31.710560 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:24:31.801721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:31.801697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.008152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:32.008086 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.183738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:32.183697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.367322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:32.367291 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.594454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:32.594372 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.807252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:32.807222 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:33.920705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:33.920672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:34.590297 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.590269 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 01:24:34.616705 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.616684 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 01:24:34.675835 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.675808 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\") pod \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " Oct 30 01:24:34.675963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.675907 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\" (UniqueName: \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\") pod \"e727a883-9c06-4c7e-a136-605055f927ae\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " Oct 30 01:24:34.675963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.675956 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\" (UniqueName: \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\") pod \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\" (UID: \"5f6e60b5-afa2-4250-b14a-f4d97f3315f3\") " Oct 30 01:24:34.676041 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.676030 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\") pod \"e727a883-9c06-4c7e-a136-605055f927ae\" (UID: \"e727a883-9c06-4c7e-a136-605055f927ae\") " Oct 30 01:24:34.676244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.676221 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume" pod "5f6e60b5-afa2-4250-b14a-f4d97f3315f3" (UID: "5f6e60b5-afa2-4250-b14a-f4d97f3315f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:34.676401 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.676368 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:34.676514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.676441 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume" pod "e727a883-9c06-4c7e-a136-605055f927ae" (UID: "e727a883-9c06-4c7e-a136-605055f927ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:34.677953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.677932 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5" pod "e727a883-9c06-4c7e-a136-605055f927ae" (UID: "e727a883-9c06-4c7e-a136-605055f927ae"). InnerVolumeSpecName "kube-api-access-rsnv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:34.678278 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.678252 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl" pod "5f6e60b5-afa2-4250-b14a-f4d97f3315f3" (UID: "5f6e60b5-afa2-4250-b14a-f4d97f3315f3"). InnerVolumeSpecName "kube-api-access-x8tbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:34.777182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.777158 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnv5\" (UniqueName: \"kubernetes.io/projected/e727a883-9c06-4c7e-a136-605055f927ae-kube-api-access-rsnv5\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:34.777182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.777180 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8tbl\" (UniqueName: \"kubernetes.io/projected/5f6e60b5-afa2-4250-b14a-f4d97f3315f3-kube-api-access-x8tbl\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:34.777312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:34.777190 5938 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e727a883-9c06-4c7e-a136-605055f927ae-plugins-config-volume\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:24:35.265831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.265797 5938 generic.go:356] "Generic (PLEG): container finished" podID="e727a883-9c06-4c7e-a136-605055f927ae" containerID="686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4" exitCode=0 Oct 30 01:24:35.266236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.265852 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" Oct 30 01:24:35.266236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.265887 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" event={"ID":"e727a883-9c06-4c7e-a136-605055f927ae","Type":"ContainerDied","Data":"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4"} Oct 30 01:24:35.266236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.265943 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24" event={"ID":"e727a883-9c06-4c7e-a136-605055f927ae","Type":"ContainerDied","Data":"3056becaec0b6d04207a589c3c0f5056180cba3a55010e24720bc7a7c3d888cb"} Oct 30 01:24:35.266236 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.265972 5938 scope.go:122] "RemoveContainer" containerID="686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4" Oct 30 01:24:35.267884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.267839 5938 generic.go:356] "Generic (PLEG): container finished" podID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerID="998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1" exitCode=0 Oct 30 01:24:35.268001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.267923 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" Oct 30 01:24:35.268001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.267963 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" event={"ID":"5f6e60b5-afa2-4250-b14a-f4d97f3315f3","Type":"ContainerDied","Data":"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1"} Oct 30 01:24:35.268091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.268028 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7" event={"ID":"5f6e60b5-afa2-4250-b14a-f4d97f3315f3","Type":"ContainerDied","Data":"f6315229ad1934f0bf09186f5d2d210bc11c5e5b2a9309d7875cbca62020827c"} Oct 30 01:24:35.281043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.281022 5938 scope.go:122] "RemoveContainer" containerID="686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4" Oct 30 01:24:35.282070 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:35.281413 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4\": container with ID starting with 686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4 not found: ID does not exist" containerID="686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4" Oct 30 01:24:35.282070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.281446 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4"} err="failed to get container status \"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4\": rpc error: code = NotFound desc = could not find container \"686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4\": container with ID starting with 686229ca244077b6a548eef48b1d4ae25228aeb87cfb201fd8a1b5ad8694aff4 not found: ID does not exist" Oct 30 01:24:35.282070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.281470 5938 scope.go:122] "RemoveContainer" containerID="998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1" Oct 30 01:24:35.291192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.291168 5938 scope.go:122] "RemoveContainer" containerID="998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1" Oct 30 01:24:35.291495 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:35.291475 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1\": container with ID starting with 998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1 not found: ID does not exist" containerID="998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1" Oct 30 01:24:35.291559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.291503 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1"} err="failed to get container status \"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1\": rpc error: code = NotFound desc = could not find container \"998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1\": container with ID starting with 998caae662f1c35f41eb417ef365a8ee1da8e6f949fc503cb2b6c5ced6fb3dc1 not found: ID does not exist" Oct 30 01:24:35.330165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.330143 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 01:24:35.341307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.341288 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-021d6d-67c98cf9c9-qkfq7"] Oct 30 01:24:35.365108 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.365083 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 01:24:35.368785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.368766 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-021d6d-765d889d96-p2p24"] Oct 30 01:24:35.876113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.876085 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" path="/var/lib/kubelet/pods/5f6e60b5-afa2-4250-b14a-f4d97f3315f3/volumes" Oct 30 01:24:35.876467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:35.876451 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e727a883-9c06-4c7e-a136-605055f927ae" path="/var/lib/kubelet/pods/e727a883-9c06-4c7e-a136-605055f927ae/volumes" Oct 30 01:24:40.193377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:40.193330 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.331667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:40.331637 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:41.626022 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:41.625983 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:41.764029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:41.763999 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:41.912948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:41.912879 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:42.060538 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:42.060506 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:42.217782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:42.217710 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:47.911369 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:24:47.911335 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:58.832925 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:24:58.832886 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:25:00.568234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.568194 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt"] Oct 30 01:25:00.569957 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.569919 5938 cpu_manager.go:414] "containerMap: removing container" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" Oct 30 01:25:00.569957 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.569950 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.569981 5938 cpu_manager.go:414] "containerMap: removing container" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.569990 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570013 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" containerName="revoke-keys" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570021 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" containerName="revoke-keys" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570070 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570078 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570087 5938 cpu_manager.go:414] "containerMap: removing container" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570095 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570125 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" Oct 30 01:25:00.570189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570134 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570312 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8a428a-e570-46b0-a82d-838d0c76cad0" containerName="main" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570349 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6e60b5-afa2-4250-b14a-f4d97f3315f3" containerName="payload-pre-processing" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570361 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" containerName="revoke-keys" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570379 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e727a883-9c06-4c7e-a136-605055f927ae" containerName="payload-processing" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570389 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe67c1fd-69f6-41a5-a599-277101dfd095" containerName="payload-processing" Oct 30 01:25:00.570707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.570398 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8094870f-eac3-4eae-922e-95a27b456eba" containerName="main" Oct 30 01:25:00.574404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.574372 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.590445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.590416 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt"] Oct 30 01:25:00.600307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.600272 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:25:00.600307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.600282 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:25:00.600466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.600338 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:25:00.600466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.600343 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:25:00.693210 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.693186 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.693339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.693231 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.693339 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.693318 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjsm\" (UniqueName: \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.794030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.794004 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.794148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.794048 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.794148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.794084 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjsm\" (UniqueName: \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.794686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.794660 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.796748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.796725 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.817035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.817005 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjsm\" (UniqueName: \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\") pod \"maas-api-revoke-sub-85237e45-rqtbt\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:00.888900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:00.888877 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:01.182641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:01.182568 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt"] Oct 30 01:25:01.187207 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:25:01.187174 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600f0507_2dcc_44e3_b07f_4d77b3c9e86f.slice/crio-9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383 WatchSource:0}: Error finding container 9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383: Status 404 returned error can't find the container with id 9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383 Oct 30 01:25:01.403390 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:01.403358 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" event={"ID":"600f0507-2dcc-44e3-b07f-4d77b3c9e86f","Type":"ContainerStarted","Data":"9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383"} Oct 30 01:25:02.410671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:02.410633 5938 generic.go:356] "Generic (PLEG): container finished" podID="600f0507-2dcc-44e3-b07f-4d77b3c9e86f" containerID="f17fc245aec81b0495f85bff41c819a481c516b43a5dcdd6a27e3231b0c294d2" exitCode=0 Oct 30 01:25:02.411054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:02.410714 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" event={"ID":"600f0507-2dcc-44e3-b07f-4d77b3c9e86f","Type":"ContainerDied","Data":"f17fc245aec81b0495f85bff41c819a481c516b43a5dcdd6a27e3231b0c294d2"} Oct 30 01:25:03.547490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.547468 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:03.720199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.720135 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\") pod \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " Oct 30 01:25:03.720199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.720180 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\" (UniqueName: \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\") pod \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " Oct 30 01:25:03.720379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.720206 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\") pod \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\" (UID: \"600f0507-2dcc-44e3-b07f-4d77b3c9e86f\") " Oct 30 01:25:03.720550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.720520 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca" pod "600f0507-2dcc-44e3-b07f-4d77b3c9e86f" (UID: "600f0507-2dcc-44e3-b07f-4d77b3c9e86f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:03.722258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.722228 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm" pod "600f0507-2dcc-44e3-b07f-4d77b3c9e86f" (UID: "600f0507-2dcc-44e3-b07f-4d77b3c9e86f"). InnerVolumeSpecName "kube-api-access-9gjsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:03.722360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.722281 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca" pod "600f0507-2dcc-44e3-b07f-4d77b3c9e86f" (UID: "600f0507-2dcc-44e3-b07f-4d77b3c9e86f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:03.821030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.821003 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.821030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.821032 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjsm\" (UniqueName: \"kubernetes.io/projected/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-kube-api-access-9gjsm\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:03.821171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:03.821042 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/600f0507-2dcc-44e3-b07f-4d77b3c9e86f-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:04.422639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.422607 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" event={"ID":"600f0507-2dcc-44e3-b07f-4d77b3c9e86f","Type":"ContainerDied","Data":"9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383"} Oct 30 01:25:04.422639 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.422628 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt" Oct 30 01:25:04.422896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.422665 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eefcbb43ec26df17b677c9ef897f9fcb53948843565422f20137e6555aaa383" Oct 30 01:25:04.941651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.941619 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4"] Oct 30 01:25:04.942077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.941992 5938 cpu_manager.go:414] "containerMap: removing container" podUID="600f0507-2dcc-44e3-b07f-4d77b3c9e86f" containerName="revoke-keys" Oct 30 01:25:04.942077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.942005 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600f0507-2dcc-44e3-b07f-4d77b3c9e86f" containerName="revoke-keys" Oct 30 01:25:04.942165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.942102 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="600f0507-2dcc-44e3-b07f-4d77b3c9e86f" containerName="revoke-keys" Oct 30 01:25:04.945136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.945113 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:04.955852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.955826 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw"] Oct 30 01:25:04.959837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.959817 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:04.965692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.965667 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:25:04.965692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.965667 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwncz\"" Oct 30 01:25:04.972170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.972136 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4"] Oct 30 01:25:04.981310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:04.981286 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw"] Oct 30 01:25:05.027732 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.027694 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20309a7-fa26-440e-80bb-955b9fac548b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.027843 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.027741 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028012 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028182 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028158 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqztw\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-kube-api-access-zqztw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028382 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028439 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028643 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028806 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20309a7-fa26-440e-80bb-955b9fac548b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.028994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.028895 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130109 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20309a7-fa26-440e-80bb-955b9fac548b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130138 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130162 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130185 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130206 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130285 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqztw\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-kube-api-access-zqztw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130308 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqs2\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-kube-api-access-fgqs2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130335 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130372 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130353 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130375 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130403 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130467 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130524 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130583 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130571 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130627 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130692 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130692 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130720 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130750 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20309a7-fa26-440e-80bb-955b9fac548b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130765 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.130852 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.130779 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.131237 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.131099 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.131286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.131267 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20309a7-fa26-440e-80bb-955b9fac548b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.132708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.132684 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20309a7-fa26-440e-80bb-955b9fac548b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.132791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.132775 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20309a7-fa26-440e-80bb-955b9fac548b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.152501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.152482 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.152588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.152530 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqztw\" (UniqueName: \"kubernetes.io/projected/b20309a7-fa26-440e-80bb-955b9fac548b-kube-api-access-zqztw\") pod \"maas-default-gateway-openshift-default-7d9897b54c-srsm4\" (UID: \"b20309a7-fa26-440e-80bb-955b9fac548b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.231736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231683 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqs2\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-kube-api-access-fgqs2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231726 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231745 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231771 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231796 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231812 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.231886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231831 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231895 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.231930 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.232133 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.232357 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.232396 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.232421 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.232573 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.232553 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.234267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.234243 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.234592 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.234570 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.253831 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.253808 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.253980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.253963 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqs2\" (UniqueName: \"kubernetes.io/projected/1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0-kube-api-access-fgqs2\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kgxxw\" (UID: \"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.259714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.259696 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:05.272987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.272968 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:05.584209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.584136 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw"] Oct 30 01:25:05.587306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.587280 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4"] Oct 30 01:25:05.587510 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:25:05.587487 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f97dbd6_ff80_40ec_bfd5_b1ed49b70ab0.slice/crio-a71fe10f2b973fca0c1c39de5d795b24907bdc417fe68de060818dfbb7f747f5 WatchSource:0}: Error finding container a71fe10f2b973fca0c1c39de5d795b24907bdc417fe68de060818dfbb7f747f5: Status 404 returned error can't find the container with id a71fe10f2b973fca0c1c39de5d795b24907bdc417fe68de060818dfbb7f747f5 Oct 30 01:25:05.589653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.589618 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:05.589761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.589681 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:05.589761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.589708 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:05.599241 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:25:05.599220 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20309a7_fa26_440e_80bb_955b9fac548b.slice/crio-7daf80d22b96d53a9091272bbddc47555c6120f8b2ccd12cdbcea39a62356bf7 WatchSource:0}: Error finding container 7daf80d22b96d53a9091272bbddc47555c6120f8b2ccd12cdbcea39a62356bf7: Status 404 returned error can't find the container with id 7daf80d22b96d53a9091272bbddc47555c6120f8b2ccd12cdbcea39a62356bf7 Oct 30 01:25:05.601360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.601330 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:05.601433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.601394 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:05.601433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:05.601423 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:25:06.441609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.441571 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" event={"ID":"b20309a7-fa26-440e-80bb-955b9fac548b","Type":"ContainerStarted","Data":"6ec6691d39ad40d328da3f66fdafa14f55acd80ea5696065377eb3035ed78f1b"} Oct 30 01:25:06.442154 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.441641 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" event={"ID":"b20309a7-fa26-440e-80bb-955b9fac548b","Type":"ContainerStarted","Data":"7daf80d22b96d53a9091272bbddc47555c6120f8b2ccd12cdbcea39a62356bf7"} Oct 30 01:25:06.443290 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.443267 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" event={"ID":"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0","Type":"ContainerStarted","Data":"296c7c65e841d3b0fe5ea122461bd5ab15d226031cce374d3dbec7c6fcd6d550"} Oct 30 01:25:06.443350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.443310 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" event={"ID":"1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0","Type":"ContainerStarted","Data":"a71fe10f2b973fca0c1c39de5d795b24907bdc417fe68de060818dfbb7f747f5"} Oct 30 01:25:06.512886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.509595 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podStartSLOduration=2.509572778 podStartE2EDuration="2.509572778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:06.507382464 +0000 UTC m=+3158.747617584" watchObservedRunningTime="2026-10-30 01:25:06.509572778 +0000 UTC m=+3158.749807889" Oct 30 01:25:06.559091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:06.559037 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podStartSLOduration=2.55902173 podStartE2EDuration="2.55902173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:06.557278634 +0000 UTC m=+3158.797513747" watchObservedRunningTime="2026-10-30 01:25:06.55902173 +0000 UTC m=+3158.799256844" Oct 30 01:25:07.260735 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.260697 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:07.262308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.262266 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:07.262471 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.262336 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:07.273666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.273640 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:07.275911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.275881 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:07.276046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:07.275932 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:08.260384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:08.260348 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:08.260952 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:08.260411 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:08.274132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:08.274108 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:08.274266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:08.274148 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:09.260932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:09.260820 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:09.261315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:09.260928 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:09.274102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:09.274070 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:09.274247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:09.274115 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:10.260637 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:10.260601 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:10.260829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:10.260651 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:10.274349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:10.274322 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:10.274661 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:10.274372 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:11.260750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:11.260714 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:11.260976 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:11.260767 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:11.273414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:11.273388 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:11.273568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:11.273426 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:12.260379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:12.260342 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:12.260785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:12.260405 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:12.274104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:12.274076 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 01:25:12.274225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:12.274126 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 01:25:12.314415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:12.314385 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:13.261021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:13.260981 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" start-of-body= Oct 30 01:25:13.261576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:13.261057 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": dial tcp 10.134.1.84:15021: connect: connection refused" Oct 30 01:25:13.997336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:13.997309 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:13.997955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:13.997760 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:13.998897 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:13.998876 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" Oct 30 01:25:14.264016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:14.263934 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:14.265810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:14.264498 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:14.265810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:14.265404 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" Oct 30 01:25:14.501514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:14.501485 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.833291 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:25:28.833251 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:25:41.879490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:41.879451 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:51.339563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.339525 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn"] Oct 30 01:25:51.344625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.344605 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.360597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.360570 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn"] Oct 30 01:25:51.370325 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.370295 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:25:51.370490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.370350 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:25:51.370580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.370383 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:25:51.370580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.370387 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:25:51.427133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.427102 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fccz\" (UniqueName: \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.427306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.427142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.427306 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.427175 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.527977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.527941 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fccz\" (UniqueName: \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.528138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.527988 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.528138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.528040 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.528646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.528621 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.530323 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.530300 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.550255 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.550231 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fccz\" (UniqueName: \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\") pod \"maas-api-revoke-sub-c1e271a9-hvfpn\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.657645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.657623 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:51.951251 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.951174 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn"] Oct 30 01:25:51.954125 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:25:51.954098 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a354e2_6547_41bb_b06c_08f8a0dbea25.slice/crio-587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb WatchSource:0}: Error finding container 587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb: Status 404 returned error can't find the container with id 587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb Oct 30 01:25:51.968546 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:51.968520 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:52.119485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:52.119460 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:52.700695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:52.700664 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" event={"ID":"60a354e2-6547-41bb-b06c-08f8a0dbea25","Type":"ContainerStarted","Data":"ec15d2317497504530052a298722bd0fb4e22408c87a6a8b8aeaca3138e98444"} Oct 30 01:25:52.701051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:52.700722 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" event={"ID":"60a354e2-6547-41bb-b06c-08f8a0dbea25","Type":"ContainerStarted","Data":"587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb"} Oct 30 01:25:52.740061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:52.740020 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" podStartSLOduration=1.740007303 podStartE2EDuration="1.740007303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:52.739030924 +0000 UTC m=+3204.979266047" watchObservedRunningTime="2026-10-30 01:25:52.740007303 +0000 UTC m=+3204.980242416" Oct 30 01:25:53.708911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:53.708846 5938 generic.go:356] "Generic (PLEG): container finished" podID="60a354e2-6547-41bb-b06c-08f8a0dbea25" containerID="ec15d2317497504530052a298722bd0fb4e22408c87a6a8b8aeaca3138e98444" exitCode=0 Oct 30 01:25:53.709364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:53.708940 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" event={"ID":"60a354e2-6547-41bb-b06c-08f8a0dbea25","Type":"ContainerDied","Data":"ec15d2317497504530052a298722bd0fb4e22408c87a6a8b8aeaca3138e98444"} Oct 30 01:25:54.850029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.850003 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:54.955417 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.955392 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\") pod \"60a354e2-6547-41bb-b06c-08f8a0dbea25\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " Oct 30 01:25:54.955587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.955504 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\" (UniqueName: \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\") pod \"60a354e2-6547-41bb-b06c-08f8a0dbea25\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " Oct 30 01:25:54.955587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.955549 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\") pod \"60a354e2-6547-41bb-b06c-08f8a0dbea25\" (UID: \"60a354e2-6547-41bb-b06c-08f8a0dbea25\") " Oct 30 01:25:54.955819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.955794 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca" pod "60a354e2-6547-41bb-b06c-08f8a0dbea25" (UID: "60a354e2-6547-41bb-b06c-08f8a0dbea25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:54.957486 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.957461 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca" pod "60a354e2-6547-41bb-b06c-08f8a0dbea25" (UID: "60a354e2-6547-41bb-b06c-08f8a0dbea25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:54.957629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:54.957615 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz" pod "60a354e2-6547-41bb-b06c-08f8a0dbea25" (UID: "60a354e2-6547-41bb-b06c-08f8a0dbea25"). InnerVolumeSpecName "kube-api-access-9fccz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:55.056904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.056835 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:55.056904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.056856 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a354e2-6547-41bb-b06c-08f8a0dbea25-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:55.056904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.056889 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fccz\" (UniqueName: \"kubernetes.io/projected/60a354e2-6547-41bb-b06c-08f8a0dbea25-kube-api-access-9fccz\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:25:55.721069 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.721040 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" Oct 30 01:25:55.721252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.721041 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn" event={"ID":"60a354e2-6547-41bb-b06c-08f8a0dbea25","Type":"ContainerDied","Data":"587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb"} Oct 30 01:25:55.721252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:25:55.721174 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="587214cdf8cad72f901a0c08306d8b1de3288fb8a11b4b2e29506e31aad38fbb" Oct 30 01:25:58.834100 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:25:58.834064 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:26:05.117795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.117754 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j"] Oct 30 01:26:05.118349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.118328 5938 cpu_manager.go:414] "containerMap: removing container" podUID="60a354e2-6547-41bb-b06c-08f8a0dbea25" containerName="revoke-keys" Oct 30 01:26:05.118448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.118351 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a354e2-6547-41bb-b06c-08f8a0dbea25" containerName="revoke-keys" Oct 30 01:26:05.118524 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.118493 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a354e2-6547-41bb-b06c-08f8a0dbea25" containerName="revoke-keys" Oct 30 01:26:05.122104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.122080 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.158622 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.152972 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j"] Oct 30 01:26:05.158622 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.155660 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb"] Oct 30 01:26:05.160043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.160010 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.177561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.177525 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb"] Oct 30 01:26:05.238986 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.238944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2491bdb-6953-40e8-8c95-5c39bca71b56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239027 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239287 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239328 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw77t\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-kube-api-access-gw77t\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239441 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239517 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239552 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239670 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbtj\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-kube-api-access-bzbtj\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239717 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.239837 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239762 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239844 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.239925 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239904 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.240064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.239941 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04691eac-59b2-40f2-bea3-84cba2e43e17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.240064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.240002 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.240064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.240050 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.240246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.240103 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.341453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341420 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.341453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341452 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341476 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341495 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341525 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw77t\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-kube-api-access-gw77t\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341557 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341581 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341612 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341657 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbtj\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-kube-api-access-bzbtj\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.341703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341687 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341708 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341739 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341767 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341799 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04691eac-59b2-40f2-bea3-84cba2e43e17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341840 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341890 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341892 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.341987 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342026 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2491bdb-6953-40e8-8c95-5c39bca71b56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342098 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342150 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342172 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342407 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342413 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342484 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.342911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342700 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2491bdb-6953-40e8-8c95-5c39bca71b56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.342911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.342884 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.343227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.343197 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04691eac-59b2-40f2-bea3-84cba2e43e17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.344302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.344281 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.344433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.344410 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.344662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.344634 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.345020 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.345002 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.364525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.364493 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbtj\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-kube-api-access-bzbtj\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.364525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.364509 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.364655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.364568 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04691eac-59b2-40f2-bea3-84cba2e43e17-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-ccb2j\" (UID: \"04691eac-59b2-40f2-bea3-84cba2e43e17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.364655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.364628 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw77t\" (UniqueName: \"kubernetes.io/projected/e2491bdb-6953-40e8-8c95-5c39bca71b56-kube-api-access-gw77t\") pod \"maas-default-gateway-openshift-default-7d9897b54c-lvppb\" (UID: \"e2491bdb-6953-40e8-8c95-5c39bca71b56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.442353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.442301 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:05.486209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.486184 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:05.754266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.754102 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j"] Oct 30 01:26:05.756632 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:26:05.756606 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04691eac_59b2_40f2_bea3_84cba2e43e17.slice/crio-f75dbe2baac5bcb8b6259ad4104cb49f5ec6c569db1c4e6055dd7865d951500f WatchSource:0}: Error finding container f75dbe2baac5bcb8b6259ad4104cb49f5ec6c569db1c4e6055dd7865d951500f: Status 404 returned error can't find the container with id f75dbe2baac5bcb8b6259ad4104cb49f5ec6c569db1c4e6055dd7865d951500f Oct 30 01:26:05.758696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.758664 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:05.758782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.758725 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:05.758782 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.758754 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:05.781295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.781271 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" event={"ID":"04691eac-59b2-40f2-bea3-84cba2e43e17","Type":"ContainerStarted","Data":"f75dbe2baac5bcb8b6259ad4104cb49f5ec6c569db1c4e6055dd7865d951500f"} Oct 30 01:26:05.804203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.804175 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb"] Oct 30 01:26:05.808797 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:26:05.808770 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2491bdb_6953_40e8_8c95_5c39bca71b56.slice/crio-62b3703a29585785647a5433eb58201a44404eec045315034215d82f0013f36e WatchSource:0}: Error finding container 62b3703a29585785647a5433eb58201a44404eec045315034215d82f0013f36e: Status 404 returned error can't find the container with id 62b3703a29585785647a5433eb58201a44404eec045315034215d82f0013f36e Oct 30 01:26:05.811145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.811107 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:05.811225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.811193 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:05.811267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:05.811237 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:26:06.820007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:06.819964 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" event={"ID":"04691eac-59b2-40f2-bea3-84cba2e43e17","Type":"ContainerStarted","Data":"f85ea54d85daae740044f4712d95393fad4b0ccc3ae02d4b5a4635cdd0401ff5"} Oct 30 01:26:06.824328 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:06.824292 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" event={"ID":"e2491bdb-6953-40e8-8c95-5c39bca71b56","Type":"ContainerStarted","Data":"088707c66aa2211fe43a452d326eb8c0344b0b0bdbe01f2e23fd26df153aea33"} Oct 30 01:26:06.824568 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:06.824551 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" event={"ID":"e2491bdb-6953-40e8-8c95-5c39bca71b56","Type":"ContainerStarted","Data":"62b3703a29585785647a5433eb58201a44404eec045315034215d82f0013f36e"} Oct 30 01:26:06.915886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:06.915490 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podStartSLOduration=1.915471194 podStartE2EDuration="1.915471194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:06.875677473 +0000 UTC m=+3219.115912586" watchObservedRunningTime="2026-10-30 01:26:06.915471194 +0000 UTC m=+3219.155706305" Oct 30 01:26:06.945282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:06.945220 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podStartSLOduration=1.945206286 podStartE2EDuration="1.945206286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:06.943978932 +0000 UTC m=+3219.184214082" watchObservedRunningTime="2026-10-30 01:26:06.945206286 +0000 UTC m=+3219.185441398" Oct 30 01:26:07.442911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.442856 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:07.444505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.444480 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:07.444572 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.444530 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:07.488149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.488122 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:07.506021 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.505993 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:07.506303 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:07.506260 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:08.446279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:08.446241 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:08.446935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:08.446896 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:08.488455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:08.488423 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:08.488733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:08.488701 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:09.443887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:09.443216 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:09.443887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:09.443274 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:09.489895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:09.489206 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:09.489895 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:09.489259 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:10.446349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:10.446311 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:10.446681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:10.446636 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:10.487314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:10.487284 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:10.487565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:10.487537 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:11.442724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:11.442688 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:11.443195 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:11.442935 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:11.491893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:11.488148 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:11.491893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:11.488198 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:12.443337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:12.443302 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" start-of-body= Oct 30 01:26:12.443761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:12.443366 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": dial tcp 10.134.1.87:15021: connect: connection refused" Oct 30 01:26:12.486840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:12.486812 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" start-of-body= Oct 30 01:26:12.486983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:12.486884 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": dial tcp 10.134.1.88:15021: connect: connection refused" Oct 30 01:26:14.444172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.444123 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:14.444676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.444199 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:14.488336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.488305 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:14.488502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.488353 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:14.558675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.558644 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp"] Oct 30 01:26:14.563140 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.563118 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.578327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.578256 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:26:14.578508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.578478 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:26:14.586089 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.586055 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvjc\" (UniqueName: \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.586215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.586109 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.586215 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.586170 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.592727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.592694 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp"] Oct 30 01:26:14.592850 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.592796 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:26:14.592980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.592901 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:26:14.687460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.687430 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvjc\" (UniqueName: \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.687646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.687473 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.687646 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.687528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.688227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.688203 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.690075 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.690054 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.710001 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.709950 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvjc\" (UniqueName: \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\") pod \"maas-api-revoke-sub-2e1f5b5f-bc8rp\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:14.880329 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:14.880303 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:15.188090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.188046 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp"] Oct 30 01:26:15.486100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.486070 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:15.487113 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.487079 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:15.529077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.528986 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:15.529077 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.529033 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:15.941884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.941830 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" event={"ID":"c8fbc932-37fc-44cf-b8b9-af36dbc44873","Type":"ContainerStarted","Data":"035f02029c7dd640e6e179f21bd369cb642d82242f6e6ed46b61e3113cb50c16"} Oct 30 01:26:15.942189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:15.942164 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" event={"ID":"c8fbc932-37fc-44cf-b8b9-af36dbc44873","Type":"ContainerStarted","Data":"df19a6f57814b5deb7cda29c90e9f7b6b8abbc01e4f3df69cce3751b600366ed"} Oct 30 01:26:16.487706 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.487663 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:16.488256 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.487728 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:16.572029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.571989 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:26:16.572190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.572041 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:16.628681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.628655 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:16.629151 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.629125 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:16.629796 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.629776 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" Oct 30 01:26:16.670157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.670129 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:16.670452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.670426 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:16.671433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.671415 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" Oct 30 01:26:16.685287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.685236 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" podStartSLOduration=2.6852237089999997 podStartE2EDuration="2.685223709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:15.992721416 +0000 UTC m=+3228.232956533" watchObservedRunningTime="2026-10-30 01:26:16.685223709 +0000 UTC m=+3228.925458853" Oct 30 01:26:16.949091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.949052 5938 generic.go:356] "Generic (PLEG): container finished" podID="c8fbc932-37fc-44cf-b8b9-af36dbc44873" containerID="035f02029c7dd640e6e179f21bd369cb642d82242f6e6ed46b61e3113cb50c16" exitCode=0 Oct 30 01:26:16.949280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:16.949137 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" event={"ID":"c8fbc932-37fc-44cf-b8b9-af36dbc44873","Type":"ContainerDied","Data":"035f02029c7dd640e6e179f21bd369cb642d82242f6e6ed46b61e3113cb50c16"} Oct 30 01:26:18.087265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.087242 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:18.125746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.125719 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\" (UniqueName: \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\") pod \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " Oct 30 01:26:18.125933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.125798 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\") pod \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " Oct 30 01:26:18.126027 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.125944 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\") pod \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\" (UID: \"c8fbc932-37fc-44cf-b8b9-af36dbc44873\") " Oct 30 01:26:18.126933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.126901 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca" pod "c8fbc932-37fc-44cf-b8b9-af36dbc44873" (UID: "c8fbc932-37fc-44cf-b8b9-af36dbc44873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:18.129079 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.129050 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc" pod "c8fbc932-37fc-44cf-b8b9-af36dbc44873" (UID: "c8fbc932-37fc-44cf-b8b9-af36dbc44873"). InnerVolumeSpecName "kube-api-access-bxvjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:18.129768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.129735 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca" pod "c8fbc932-37fc-44cf-b8b9-af36dbc44873" (UID: "c8fbc932-37fc-44cf-b8b9-af36dbc44873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:18.227717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.227639 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvjc\" (UniqueName: \"kubernetes.io/projected/c8fbc932-37fc-44cf-b8b9-af36dbc44873-kube-api-access-bxvjc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.227717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.227664 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.227717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.227673 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fbc932-37fc-44cf-b8b9-af36dbc44873-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:18.960896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.960852 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" Oct 30 01:26:18.961052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.960892 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp" event={"ID":"c8fbc932-37fc-44cf-b8b9-af36dbc44873","Type":"ContainerDied","Data":"df19a6f57814b5deb7cda29c90e9f7b6b8abbc01e4f3df69cce3751b600366ed"} Oct 30 01:26:18.961052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:18.960936 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df19a6f57814b5deb7cda29c90e9f7b6b8abbc01e4f3df69cce3751b600366ed" Oct 30 01:26:25.764697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.764655 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6"] Oct 30 01:26:25.765335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.765312 5938 cpu_manager.go:414] "containerMap: removing container" podUID="c8fbc932-37fc-44cf-b8b9-af36dbc44873" containerName="revoke-keys" Oct 30 01:26:25.765383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.765341 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fbc932-37fc-44cf-b8b9-af36dbc44873" containerName="revoke-keys" Oct 30 01:26:25.765541 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.765526 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fbc932-37fc-44cf-b8b9-af36dbc44873" containerName="revoke-keys" Oct 30 01:26:25.769333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.769316 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.784586 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.784557 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:26:25.787432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.787402 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6"] Oct 30 01:26:25.796294 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.796272 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:26:25.796406 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.796282 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:26:25.884679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.884653 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:26:25.888042 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.888019 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.888126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.888098 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.888183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.888163 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zhv\" (UniqueName: \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.988855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.988824 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.989058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.989022 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zhv\" (UniqueName: \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.989157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.989087 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.989552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.989529 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:25.991248 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:25.991224 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:26.011025 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:26.010999 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zhv\" (UniqueName: \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\") pod \"maas-api-revoke-sub-ec843018-rf6d6\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:26.085039 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:26.084991 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:26.383189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:26.383162 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6"] Oct 30 01:26:26.384780 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:26:26.384747 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f73258_6ff6_4b8a_9177_bc5ec5c795b4.slice/crio-a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f WatchSource:0}: Error finding container a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f: Status 404 returned error can't find the container with id a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f Oct 30 01:26:27.007824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:27.007590 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" event={"ID":"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4","Type":"ContainerStarted","Data":"0b55d321c62d735210bbe85975b03e1ce556edeec4e28243c59c93c50371cb3c"} Oct 30 01:26:27.007824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:27.007663 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" event={"ID":"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4","Type":"ContainerStarted","Data":"a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f"} Oct 30 01:26:27.051209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:27.051150 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" podStartSLOduration=2.051131812 podStartE2EDuration="2.051131812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:27.048259331 +0000 UTC m=+3239.288494446" watchObservedRunningTime="2026-10-30 01:26:27.051131812 +0000 UTC m=+3239.291366926" Oct 30 01:26:28.019908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:28.019624 5938 generic.go:356] "Generic (PLEG): container finished" podID="b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" containerID="0b55d321c62d735210bbe85975b03e1ce556edeec4e28243c59c93c50371cb3c" exitCode=0 Oct 30 01:26:28.019908 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:28.019702 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" event={"ID":"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4","Type":"ContainerDied","Data":"0b55d321c62d735210bbe85975b03e1ce556edeec4e28243c59c93c50371cb3c"} Oct 30 01:26:28.839385 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:26:28.839328 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:26:29.212048 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.212016 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:29.318147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.318114 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\") pod \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " Oct 30 01:26:29.318331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.318194 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\" (UniqueName: \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\") pod \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " Oct 30 01:26:29.318331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.318282 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\") pod \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\" (UID: \"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4\") " Oct 30 01:26:29.318940 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.318912 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca" pod "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" (UID: "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:29.325502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.321168 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv" pod "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" (UID: "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4"). InnerVolumeSpecName "kube-api-access-f6zhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:29.325502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.321328 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca" pod "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" (UID: "b8f73258-6ff6-4b8a-9177-bc5ec5c795b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:29.419311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.419240 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:29.419311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.419272 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zhv\" (UniqueName: \"kubernetes.io/projected/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-kube-api-access-f6zhv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:29.419311 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:29.419287 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:30.075010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:30.062435 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" event={"ID":"b8f73258-6ff6-4b8a-9177-bc5ec5c795b4","Type":"ContainerDied","Data":"a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f"} Oct 30 01:26:30.075010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:30.062501 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a245162107a3326c0de6e5a862efa9b17bed09acb92c013269507911e9d3152f" Oct 30 01:26:30.075010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:30.062576 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6" Oct 30 01:26:42.004740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:42.004632 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:53.647953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.647920 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r"] Oct 30 01:26:53.648371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.648284 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" containerName="revoke-keys" Oct 30 01:26:53.648371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.648295 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" containerName="revoke-keys" Oct 30 01:26:53.648461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.648381 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" containerName="revoke-keys" Oct 30 01:26:53.651859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.651836 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.667427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.667402 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:26:53.679051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.679018 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:26:53.679177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.679099 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:26:53.679628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.679607 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:26:53.681258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.681235 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r"] Oct 30 01:26:53.723708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.723672 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpqb\" (UniqueName: \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.723820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.723791 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.723966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.723944 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.824377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.824350 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.824505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.824393 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpqb\" (UniqueName: \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.824576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.824521 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.825086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.825067 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.826820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.826802 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.847139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.847118 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpqb\" (UniqueName: \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\") pod \"maas-api-revoke-sub-046e1343-m2v8r\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:53.964763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:53.964710 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:54.263889 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:54.263711 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r"] Oct 30 01:26:54.266577 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:26:54.266543 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355fae7e_f48f_42c7_bc89_312fa052e561.slice/crio-f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e WatchSource:0}: Error finding container f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e: Status 404 returned error can't find the container with id f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e Oct 30 01:26:55.238630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:55.238600 5938 generic.go:356] "Generic (PLEG): container finished" podID="355fae7e-f48f-42c7-bc89-312fa052e561" containerID="a9f6fe64d8815c3982d6ba85ca023c3fd33057d3ee4b0fdc55b498967bce5c84" exitCode=0 Oct 30 01:26:55.239012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:55.238676 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" event={"ID":"355fae7e-f48f-42c7-bc89-312fa052e561","Type":"ContainerDied","Data":"a9f6fe64d8815c3982d6ba85ca023c3fd33057d3ee4b0fdc55b498967bce5c84"} Oct 30 01:26:55.239012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:55.238721 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" event={"ID":"355fae7e-f48f-42c7-bc89-312fa052e561","Type":"ContainerStarted","Data":"f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e"} Oct 30 01:26:56.390565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.390538 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:56.443655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.443628 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\") pod \"355fae7e-f48f-42c7-bc89-312fa052e561\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " Oct 30 01:26:56.443655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.443664 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\" (UniqueName: \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\") pod \"355fae7e-f48f-42c7-bc89-312fa052e561\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " Oct 30 01:26:56.443935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.443691 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\") pod \"355fae7e-f48f-42c7-bc89-312fa052e561\" (UID: \"355fae7e-f48f-42c7-bc89-312fa052e561\") " Oct 30 01:26:56.444110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.444083 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca" pod "355fae7e-f48f-42c7-bc89-312fa052e561" (UID: "355fae7e-f48f-42c7-bc89-312fa052e561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:56.445750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.445719 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb" pod "355fae7e-f48f-42c7-bc89-312fa052e561" (UID: "355fae7e-f48f-42c7-bc89-312fa052e561"). InnerVolumeSpecName "kube-api-access-zkpqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:56.445750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.445728 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca" pod "355fae7e-f48f-42c7-bc89-312fa052e561" (UID: "355fae7e-f48f-42c7-bc89-312fa052e561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:56.544536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.544481 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:56.544536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.544504 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpqb\" (UniqueName: \"kubernetes.io/projected/355fae7e-f48f-42c7-bc89-312fa052e561-kube-api-access-zkpqb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:56.544536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:56.544514 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355fae7e-f48f-42c7-bc89-312fa052e561-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:26:57.258882 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:57.258837 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" Oct 30 01:26:57.259030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:57.258888 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r" event={"ID":"355fae7e-f48f-42c7-bc89-312fa052e561","Type":"ContainerDied","Data":"f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e"} Oct 30 01:26:57.259030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:26:57.258937 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3f96c8f328ebb916c41feff0bc1ae6203f3906e67cc1f434d0ddc6e2124464e" Oct 30 01:26:58.833161 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:26:58.833125 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:27:04.091287 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.091255 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj"] Oct 30 01:27:04.091901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.091879 5938 cpu_manager.go:414] "containerMap: removing container" podUID="355fae7e-f48f-42c7-bc89-312fa052e561" containerName="revoke-keys" Oct 30 01:27:04.091994 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.091901 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355fae7e-f48f-42c7-bc89-312fa052e561" containerName="revoke-keys" Oct 30 01:27:04.092056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.091993 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="355fae7e-f48f-42c7-bc89-312fa052e561" containerName="revoke-keys" Oct 30 01:27:04.094964 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.094938 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.121456 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.121429 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:04.121617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.121474 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:27:04.121676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.121475 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:27:04.121676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.121543 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:27:04.123788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.123766 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj"] Oct 30 01:27:04.213312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.213281 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.213443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.213325 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.213443 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.213407 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9n8n\" (UniqueName: \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.314666 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.314630 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.314803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.314689 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.314803 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.314766 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9n8n\" (UniqueName: \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.315400 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.315381 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.316984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.316968 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.336786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.336755 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9n8n\" (UniqueName: \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\") pod \"maas-api-revoke-sub-d3fba056-5qbmj\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.407768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.407745 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:04.705753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.705729 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj"] Oct 30 01:27:04.708470 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:27:04.708442 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfb8496_37d7_4e03_8757_70001bfce1bf.slice/crio-b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73 WatchSource:0}: Error finding container b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73: Status 404 returned error can't find the container with id b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73 Oct 30 01:27:04.755545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:04.755524 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:05.316193 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:05.316152 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" event={"ID":"0bfb8496-37d7-4e03-8757-70001bfce1bf","Type":"ContainerStarted","Data":"43607cf6aac232748757d3d59be2881a617e78fbc040b79f6b1713cc365b5aa6"} Oct 30 01:27:05.316770 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:05.316219 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" event={"ID":"0bfb8496-37d7-4e03-8757-70001bfce1bf","Type":"ContainerStarted","Data":"b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73"} Oct 30 01:27:05.355004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:05.354705 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" podStartSLOduration=1.354685125 podStartE2EDuration="1.354685125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:05.354118345 +0000 UTC m=+3277.594353483" watchObservedRunningTime="2026-10-30 01:27:05.354685125 +0000 UTC m=+3277.594920253" Oct 30 01:27:06.363930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:06.363895 5938 generic.go:356] "Generic (PLEG): container finished" podID="0bfb8496-37d7-4e03-8757-70001bfce1bf" containerID="43607cf6aac232748757d3d59be2881a617e78fbc040b79f6b1713cc365b5aa6" exitCode=0 Oct 30 01:27:06.364533 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:06.364359 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" event={"ID":"0bfb8496-37d7-4e03-8757-70001bfce1bf","Type":"ContainerDied","Data":"43607cf6aac232748757d3d59be2881a617e78fbc040b79f6b1713cc365b5aa6"} Oct 30 01:27:08.039933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.039529 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:08.118883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.118510 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\") pod \"0bfb8496-37d7-4e03-8757-70001bfce1bf\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " Oct 30 01:27:08.118883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.118625 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\") pod \"0bfb8496-37d7-4e03-8757-70001bfce1bf\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " Oct 30 01:27:08.118883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.118683 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\" (UniqueName: \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\") pod \"0bfb8496-37d7-4e03-8757-70001bfce1bf\" (UID: \"0bfb8496-37d7-4e03-8757-70001bfce1bf\") " Oct 30 01:27:08.119123 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.118944 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca" pod "0bfb8496-37d7-4e03-8757-70001bfce1bf" (UID: "0bfb8496-37d7-4e03-8757-70001bfce1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:08.119171 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.119138 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:08.121597 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.121536 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n" pod "0bfb8496-37d7-4e03-8757-70001bfce1bf" (UID: "0bfb8496-37d7-4e03-8757-70001bfce1bf"). InnerVolumeSpecName "kube-api-access-k9n8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:08.121937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.121911 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca" pod "0bfb8496-37d7-4e03-8757-70001bfce1bf" (UID: "0bfb8496-37d7-4e03-8757-70001bfce1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:08.220259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.220220 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb8496-37d7-4e03-8757-70001bfce1bf-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:08.220259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.220250 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9n8n\" (UniqueName: \"kubernetes.io/projected/0bfb8496-37d7-4e03-8757-70001bfce1bf-kube-api-access-k9n8n\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:08.327570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.327494 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:08.398887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.398821 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" event={"ID":"0bfb8496-37d7-4e03-8757-70001bfce1bf","Type":"ContainerDied","Data":"b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73"} Oct 30 01:27:08.398887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.398856 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj" Oct 30 01:27:08.399127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.398942 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b585c093a4518ed68984f2b5392a0783dc46470f68014c057ccf161e92edfd73" Oct 30 01:27:08.421277 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:27:08.421248 5938 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfb8496_37d7_4e03_8757_70001bfce1bf.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:27:08.480838 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:08.480789 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:23.026715 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:23.026682 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f"] Oct 30 01:27:23.037813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:23.037778 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f018d58-rsn6f"] Oct 30 01:27:23.874244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:23.874202 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cde3538-d4ae-4f83-9a8f-2341b80bf36b" path="/var/lib/kubelet/pods/0cde3538-d4ae-4f83-9a8f-2341b80bf36b/volumes" Oct 30 01:27:26.823508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.823474 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl"] Oct 30 01:27:26.823936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.823875 5938 cpu_manager.go:414] "containerMap: removing container" podUID="0bfb8496-37d7-4e03-8757-70001bfce1bf" containerName="revoke-keys" Oct 30 01:27:26.823936 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.823888 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfb8496-37d7-4e03-8757-70001bfce1bf" containerName="revoke-keys" Oct 30 01:27:26.824024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.823964 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfb8496-37d7-4e03-8757-70001bfce1bf" containerName="revoke-keys" Oct 30 01:27:26.828044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.828017 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:26.843455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.843428 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:27:26.855012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.854991 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:26.855012 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.855014 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:27:26.855219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.855095 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:27:26.857036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.857015 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl"] Oct 30 01:27:26.936876 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.936831 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:26.937003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.936893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:26.937060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:26.937022 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbjv\" (UniqueName: \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.038023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.038000 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbjv\" (UniqueName: \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.038148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.038042 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.038148 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.038064 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.038689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.038669 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.040392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.040375 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.060308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.060283 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbjv\" (UniqueName: \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\") pod \"maas-api-revoke-sub-ae594979-24bpl\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.149565 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.149542 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:27.443183 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.443160 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl"] Oct 30 01:27:27.445787 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:27:27.445758 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954d7640_4f34_44f2_8e6e_d11574988cf5.slice/crio-ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96 WatchSource:0}: Error finding container ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96: Status 404 returned error can't find the container with id ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96 Oct 30 01:27:27.561198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:27.561173 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" event={"ID":"954d7640-4f34-44f2-8e6e-d11574988cf5","Type":"ContainerStarted","Data":"ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96"} Oct 30 01:27:28.568701 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:28.568626 5938 generic.go:356] "Generic (PLEG): container finished" podID="954d7640-4f34-44f2-8e6e-d11574988cf5" containerID="6d24d597c06442e88bbbbdf60ab62e764d7f2a6cf70d868232aa597257536f13" exitCode=0 Oct 30 01:27:28.569132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:28.568715 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" event={"ID":"954d7640-4f34-44f2-8e6e-d11574988cf5","Type":"ContainerDied","Data":"6d24d597c06442e88bbbbdf60ab62e764d7f2a6cf70d868232aa597257536f13"} Oct 30 01:27:28.833717 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:27:28.833637 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:27:29.712534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.712512 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:29.760674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.760647 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\") pod \"954d7640-4f34-44f2-8e6e-d11574988cf5\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " Oct 30 01:27:29.760821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.760702 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\") pod \"954d7640-4f34-44f2-8e6e-d11574988cf5\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " Oct 30 01:27:29.760821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.760791 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\" (UniqueName: \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\") pod \"954d7640-4f34-44f2-8e6e-d11574988cf5\" (UID: \"954d7640-4f34-44f2-8e6e-d11574988cf5\") " Oct 30 01:27:29.761029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.761004 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca" pod "954d7640-4f34-44f2-8e6e-d11574988cf5" (UID: "954d7640-4f34-44f2-8e6e-d11574988cf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:29.761158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.761137 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:29.762761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.762730 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca" pod "954d7640-4f34-44f2-8e6e-d11574988cf5" (UID: "954d7640-4f34-44f2-8e6e-d11574988cf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:29.762917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.762891 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv" pod "954d7640-4f34-44f2-8e6e-d11574988cf5" (UID: "954d7640-4f34-44f2-8e6e-d11574988cf5"). InnerVolumeSpecName "kube-api-access-mpbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:29.862609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.862582 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/954d7640-4f34-44f2-8e6e-d11574988cf5-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:29.862711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:29.862614 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbjv\" (UniqueName: \"kubernetes.io/projected/954d7640-4f34-44f2-8e6e-d11574988cf5-kube-api-access-mpbjv\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:30.582133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:30.582106 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" Oct 30 01:27:30.582273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:30.582141 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl" event={"ID":"954d7640-4f34-44f2-8e6e-d11574988cf5","Type":"ContainerDied","Data":"ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96"} Oct 30 01:27:30.582273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:30.582194 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc5d8877e794dd6ed8b22181736975921f6dee3827ef1906274f9b7f4494d96" Oct 30 01:27:30.697881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:30.697843 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:27:30.739727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:30.739708 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:27:32.294804 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:32.294772 5938 scope.go:122] "RemoveContainer" containerID="3ea4217d8ec6e5b8a9dd785a92b5622d0961f087d13cd6aee4dc6251ec3fa033" Oct 30 01:27:32.305432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:32.305411 5938 scope.go:122] "RemoveContainer" containerID="7a23719242bd429b6719a3a46a4580cd4bce6fe1eb4d35cc9f35b4d38615a8ae" Oct 30 01:27:42.219888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:42.219310 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:42.384887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:42.384598 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:48.498293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.498255 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g"] Oct 30 01:27:48.498898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.498815 5938 cpu_manager.go:414] "containerMap: removing container" podUID="954d7640-4f34-44f2-8e6e-d11574988cf5" containerName="revoke-keys" Oct 30 01:27:48.498898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.498836 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954d7640-4f34-44f2-8e6e-d11574988cf5" containerName="revoke-keys" Oct 30 01:27:48.499060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.498967 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="954d7640-4f34-44f2-8e6e-d11574988cf5" containerName="revoke-keys" Oct 30 01:27:48.504071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.504047 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.520050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.520022 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g"] Oct 30 01:27:48.528917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.528855 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:27:48.528917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.528888 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:48.528917 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.528881 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:27:48.529736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.529711 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:27:48.601073 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.601047 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvw7\" (UniqueName: \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.601225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.601087 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.601286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.601217 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.701824 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.701795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvw7\" (UniqueName: \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.702005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.701852 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.702005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.701980 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.702580 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.702563 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.704507 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.704478 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.724312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.724287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvw7\" (UniqueName: \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\") pod \"maas-api-revoke-sub-a4aef7d7-n8f2g\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:48.817577 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:48.817527 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:49.112429 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:49.112402 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g"] Oct 30 01:27:49.115052 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:27:49.115026 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a61fe7_7fff_4d4c_858a_977d28e7dcf1.slice/crio-7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79 WatchSource:0}: Error finding container 7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79: Status 404 returned error can't find the container with id 7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79 Oct 30 01:27:49.712591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:49.712525 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" event={"ID":"36a61fe7-7fff-4d4c-858a-977d28e7dcf1","Type":"ContainerStarted","Data":"d8749420c71670b67f4fd08faea3cf4976f8c865bf3257b2ba38d56782368c47"} Oct 30 01:27:49.712591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:49.712583 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" event={"ID":"36a61fe7-7fff-4d4c-858a-977d28e7dcf1","Type":"ContainerStarted","Data":"7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79"} Oct 30 01:27:49.751620 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:49.751565 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" podStartSLOduration=1.751548216 podStartE2EDuration="1.751548216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:49.75082272 +0000 UTC m=+3321.991057834" watchObservedRunningTime="2026-10-30 01:27:49.751548216 +0000 UTC m=+3321.991783330" Oct 30 01:27:50.719594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:50.719560 5938 generic.go:356] "Generic (PLEG): container finished" podID="36a61fe7-7fff-4d4c-858a-977d28e7dcf1" containerID="d8749420c71670b67f4fd08faea3cf4976f8c865bf3257b2ba38d56782368c47" exitCode=0 Oct 30 01:27:50.720051 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:50.719653 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" event={"ID":"36a61fe7-7fff-4d4c-858a-977d28e7dcf1","Type":"ContainerDied","Data":"d8749420c71670b67f4fd08faea3cf4976f8c865bf3257b2ba38d56782368c47"} Oct 30 01:27:51.855610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.855584 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:51.937234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.937207 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\") pod \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " Oct 30 01:27:51.937373 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.937282 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\" (UniqueName: \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\") pod \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " Oct 30 01:27:51.937482 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.937465 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\") pod \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\" (UID: \"36a61fe7-7fff-4d4c-858a-977d28e7dcf1\") " Oct 30 01:27:51.937854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.937825 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca" pod "36a61fe7-7fff-4d4c-858a-977d28e7dcf1" (UID: "36a61fe7-7fff-4d4c-858a-977d28e7dcf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:51.939295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.939274 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7" pod "36a61fe7-7fff-4d4c-858a-977d28e7dcf1" (UID: "36a61fe7-7fff-4d4c-858a-977d28e7dcf1"). InnerVolumeSpecName "kube-api-access-fqvw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:51.939385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:51.939346 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca" pod "36a61fe7-7fff-4d4c-858a-977d28e7dcf1" (UID: "36a61fe7-7fff-4d4c-858a-977d28e7dcf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:52.039186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.039130 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvw7\" (UniqueName: \"kubernetes.io/projected/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-kube-api-access-fqvw7\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.039186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.039153 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.039186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.039164 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a61fe7-7fff-4d4c-858a-977d28e7dcf1-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.732258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.732225 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" Oct 30 01:27:52.732440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.732255 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g" event={"ID":"36a61fe7-7fff-4d4c-858a-977d28e7dcf1","Type":"ContainerDied","Data":"7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79"} Oct 30 01:27:52.732440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:27:52.732307 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7695b570545f7c9f518ea2a2207c84751f5c6e35128eb465d2c68a2547d8ba79" Oct 30 01:27:58.833538 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:27:58.833501 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:28:04.026944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:04.026912 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp"] Oct 30 01:28:04.038809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:04.038779 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b"] Oct 30 01:28:04.040469 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:04.040442 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-918f7a86-b5ckp"] Oct 30 01:28:04.041890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:04.041857 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a0a5a-xpz5b"] Oct 30 01:28:05.877600 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:05.877561 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2ebccf-3746-4731-9935-efbc9f4ab00c" path="/var/lib/kubelet/pods/3f2ebccf-3746-4731-9935-efbc9f4ab00c/volumes" Oct 30 01:28:05.878186 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:05.878163 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ea663f-75a1-4abb-bb7b-c94702adf2d4" path="/var/lib/kubelet/pods/69ea663f-75a1-4abb-bb7b-c94702adf2d4/volumes" Oct 30 01:28:09.148931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.148807 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg"] Oct 30 01:28:09.149536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.149422 5938 cpu_manager.go:414] "containerMap: removing container" podUID="36a61fe7-7fff-4d4c-858a-977d28e7dcf1" containerName="revoke-keys" Oct 30 01:28:09.149536 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.149446 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a61fe7-7fff-4d4c-858a-977d28e7dcf1" containerName="revoke-keys" Oct 30 01:28:09.149679 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.149592 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a61fe7-7fff-4d4c-858a-977d28e7dcf1" containerName="revoke-keys" Oct 30 01:28:09.153172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.153147 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.167588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.167534 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:09.167588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.167551 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:09.179229 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.179208 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:28:09.179355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.179241 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:09.182499 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.182464 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg"] Oct 30 01:28:09.233332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.233307 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.233450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.233372 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkwk\" (UniqueName: \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.233501 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.233477 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.334031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.334008 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.334149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.334064 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkwk\" (UniqueName: \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.334149 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.334103 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.334626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.334606 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.336302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.336283 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.356267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.356238 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkwk\" (UniqueName: \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\") pod \"maas-api-revoke-sub-79805ca9-x22kg\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.467017 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.466965 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:09.763813 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.763791 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg"] Oct 30 01:28:09.766488 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:28:09.766458 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790e2615_f73c_4564_9d90_f8df4ffaaa73.slice/crio-90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b WatchSource:0}: Error finding container 90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b: Status 404 returned error can't find the container with id 90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b Oct 30 01:28:09.866910 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:09.866858 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" event={"ID":"790e2615-f73c-4564-9d90-f8df4ffaaa73","Type":"ContainerStarted","Data":"90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b"} Oct 30 01:28:10.874855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:10.874779 5938 generic.go:356] "Generic (PLEG): container finished" podID="790e2615-f73c-4564-9d90-f8df4ffaaa73" containerID="ab928338d2cefc73a445205e69500ae4aaf1650efb35cd3499ba7d46cf95c52a" exitCode=0 Oct 30 01:28:10.875270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:10.874849 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" event={"ID":"790e2615-f73c-4564-9d90-f8df4ffaaa73","Type":"ContainerDied","Data":"ab928338d2cefc73a445205e69500ae4aaf1650efb35cd3499ba7d46cf95c52a"} Oct 30 01:28:12.021043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.021020 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:12.159660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.159593 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\" (UniqueName: \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\") pod \"790e2615-f73c-4564-9d90-f8df4ffaaa73\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " Oct 30 01:28:12.159660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.159639 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\") pod \"790e2615-f73c-4564-9d90-f8df4ffaaa73\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " Oct 30 01:28:12.159904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.159717 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\") pod \"790e2615-f73c-4564-9d90-f8df4ffaaa73\" (UID: \"790e2615-f73c-4564-9d90-f8df4ffaaa73\") " Oct 30 01:28:12.160165 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.160138 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca" pod "790e2615-f73c-4564-9d90-f8df4ffaaa73" (UID: "790e2615-f73c-4564-9d90-f8df4ffaaa73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:12.161836 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.161816 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk" pod "790e2615-f73c-4564-9d90-f8df4ffaaa73" (UID: "790e2615-f73c-4564-9d90-f8df4ffaaa73"). InnerVolumeSpecName "kube-api-access-jwkwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:12.161951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.161931 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca" pod "790e2615-f73c-4564-9d90-f8df4ffaaa73" (UID: "790e2615-f73c-4564-9d90-f8df4ffaaa73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:12.260393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.260369 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkwk\" (UniqueName: \"kubernetes.io/projected/790e2615-f73c-4564-9d90-f8df4ffaaa73-kube-api-access-jwkwk\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.260393 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.260390 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.260526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.260400 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790e2615-f73c-4564-9d90-f8df4ffaaa73-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.888070 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.888047 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" Oct 30 01:28:12.888220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.888068 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg" event={"ID":"790e2615-f73c-4564-9d90-f8df4ffaaa73","Type":"ContainerDied","Data":"90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b"} Oct 30 01:28:12.888220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:12.888113 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ee0e0e834b566ad9cab5dda4b8a4b4293f5b54f9fd81d8412106343c15fc9b" Oct 30 01:28:14.027817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:14.027780 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5"] Oct 30 01:28:14.039784 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:14.039750 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd"] Oct 30 01:28:14.041280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:14.041259 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3becc40-fd6q5"] Oct 30 01:28:14.042688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:14.042664 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584d0bc3-mmpcd"] Oct 30 01:28:15.874109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:15.874078 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9397cf03-fcfd-42b2-ac44-e17e36230e9b" path="/var/lib/kubelet/pods/9397cf03-fcfd-42b2-ac44-e17e36230e9b/volumes" Oct 30 01:28:15.874574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:15.874468 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ecbc2e-b8ab-4e40-8ba5-e58acde21371" path="/var/lib/kubelet/pods/e6ecbc2e-b8ab-4e40-8ba5-e58acde21371/volumes" Oct 30 01:28:25.984749 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:25.984701 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg"] Oct 30 01:28:25.985385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:25.985359 5938 cpu_manager.go:414] "containerMap: removing container" podUID="790e2615-f73c-4564-9d90-f8df4ffaaa73" containerName="revoke-keys" Oct 30 01:28:25.985468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:25.985388 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790e2615-f73c-4564-9d90-f8df4ffaaa73" containerName="revoke-keys" Oct 30 01:28:25.985527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:25.985490 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="790e2615-f73c-4564-9d90-f8df4ffaaa73" containerName="revoke-keys" Oct 30 01:28:25.989670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:25.989646 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.016351 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.016324 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:26.016535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.016338 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:26.016535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.016353 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:26.016699 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.016389 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:28:26.018461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.018439 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg"] Oct 30 01:28:26.105295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.105272 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.105459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.105314 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcvr\" (UniqueName: \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.105459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.105440 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.206498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.206474 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.206609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.206509 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcvr\" (UniqueName: \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.206609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.206561 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.207199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.207177 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.208880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.208850 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.229266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.229242 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcvr\" (UniqueName: \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\") pod \"maas-api-revoke-sub-0b7e8edb-qmlqg\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.303434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.303379 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:26.598208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:26.598178 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg"] Oct 30 01:28:26.603815 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:28:26.603786 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3460fc9_e075_4df7_983a_7a3c30ec199e.slice/crio-0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b WatchSource:0}: Error finding container 0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b: Status 404 returned error can't find the container with id 0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b Oct 30 01:28:27.002211 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:27.002169 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" event={"ID":"b3460fc9-e075-4df7-983a-7a3c30ec199e","Type":"ContainerStarted","Data":"dd9fd5247a22ab5d69e2955d0d3f1035cff105cbbe7c5c747896d91a723e916a"} Oct 30 01:28:27.002552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:27.002238 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" event={"ID":"b3460fc9-e075-4df7-983a-7a3c30ec199e","Type":"ContainerStarted","Data":"0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b"} Oct 30 01:28:27.049114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:27.049060 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" podStartSLOduration=2.049045617 podStartE2EDuration="2.049045617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:27.048172511 +0000 UTC m=+3359.288407624" watchObservedRunningTime="2026-10-30 01:28:27.049045617 +0000 UTC m=+3359.289280730" Oct 30 01:28:28.010104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:28.010071 5938 generic.go:356] "Generic (PLEG): container finished" podID="b3460fc9-e075-4df7-983a-7a3c30ec199e" containerID="dd9fd5247a22ab5d69e2955d0d3f1035cff105cbbe7c5c747896d91a723e916a" exitCode=0 Oct 30 01:28:28.010631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:28.010440 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" event={"ID":"b3460fc9-e075-4df7-983a-7a3c30ec199e","Type":"ContainerDied","Data":"dd9fd5247a22ab5d69e2955d0d3f1035cff105cbbe7c5c747896d91a723e916a"} Oct 30 01:28:28.834649 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:28:28.834613 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:28:29.145267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.145240 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:29.232628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.232601 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\") pod \"b3460fc9-e075-4df7-983a-7a3c30ec199e\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " Oct 30 01:28:29.232771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.232644 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\" (UniqueName: \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\") pod \"b3460fc9-e075-4df7-983a-7a3c30ec199e\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " Oct 30 01:28:29.232771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.232722 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\") pod \"b3460fc9-e075-4df7-983a-7a3c30ec199e\" (UID: \"b3460fc9-e075-4df7-983a-7a3c30ec199e\") " Oct 30 01:28:29.233117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.233087 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca" pod "b3460fc9-e075-4df7-983a-7a3c30ec199e" (UID: "b3460fc9-e075-4df7-983a-7a3c30ec199e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:29.234526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.234502 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca" pod "b3460fc9-e075-4df7-983a-7a3c30ec199e" (UID: "b3460fc9-e075-4df7-983a-7a3c30ec199e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:29.234617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.234602 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr" pod "b3460fc9-e075-4df7-983a-7a3c30ec199e" (UID: "b3460fc9-e075-4df7-983a-7a3c30ec199e"). InnerVolumeSpecName "kube-api-access-hmcvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:29.334056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.334033 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:29.334056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.334054 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcvr\" (UniqueName: \"kubernetes.io/projected/b3460fc9-e075-4df7-983a-7a3c30ec199e-kube-api-access-hmcvr\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:29.334189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:29.334063 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3460fc9-e075-4df7-983a-7a3c30ec199e-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:30.024392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:30.024366 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" Oct 30 01:28:30.024392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:30.024360 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg" event={"ID":"b3460fc9-e075-4df7-983a-7a3c30ec199e","Type":"ContainerDied","Data":"0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b"} Oct 30 01:28:30.024571 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:30.024421 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0662359e0305c33fb163fe56bd9d2d12188380da428f97d1a2393d1e7e9b569b" Oct 30 01:28:32.329878 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:32.329830 5938 scope.go:122] "RemoveContainer" containerID="16cf12f37217f0b7143fa732f0dbf03067abaa808f7e1c9146e379f270bdbf57" Oct 30 01:28:32.339246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:32.339227 5938 scope.go:122] "RemoveContainer" containerID="a73a2e9ca2de6e4e3be671860312863567093be93d73cfc84de235bac5c0d1db" Oct 30 01:28:32.347273 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:32.347253 5938 scope.go:122] "RemoveContainer" containerID="aaf5372b1bd1ab7412d615037e6d88c1bb862f585f18e382bbedc6f2b98f8d37" Oct 30 01:28:32.355295 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:32.355276 5938 scope.go:122] "RemoveContainer" containerID="419dd5d7bfb7b3f5371be9e8d1d0777049f613a40b359f3525143aa364663c9d" Oct 30 01:28:32.363003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:32.362979 5938 scope.go:122] "RemoveContainer" containerID="5b4aba898309914c8378c8137d5cb76502d066fdeb151760d19116a00775e092" Oct 30 01:28:36.751046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.751014 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp"] Oct 30 01:28:36.751460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.751412 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b3460fc9-e075-4df7-983a-7a3c30ec199e" containerName="revoke-keys" Oct 30 01:28:36.751460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.751425 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3460fc9-e075-4df7-983a-7a3c30ec199e" containerName="revoke-keys" Oct 30 01:28:36.751557 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.751516 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3460fc9-e075-4df7-983a-7a3c30ec199e" containerName="revoke-keys" Oct 30 01:28:36.756310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.756291 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.783338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.783312 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:36.783338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.783312 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:36.783588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.783317 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:36.783588 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.783409 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:28:36.785381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.785358 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp"] Oct 30 01:28:36.795078 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.795049 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qr4\" (UniqueName: \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.795197 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.795096 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.795342 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.795320 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.896738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.896697 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.896951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.896786 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qr4\" (UniqueName: \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.896951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.896819 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.897405 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.897377 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.899096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.899075 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:36.918907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:36.918890 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qr4\" (UniqueName: \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\") pod \"maas-api-revoke-sub-3d4156f1-ttjqp\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:37.069894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:37.069816 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:37.367181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:37.366974 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp"] Oct 30 01:28:37.369936 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:28:37.369904 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dab4abc_8b9f_4c49_b1d3_3c1593504be6.slice/crio-a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db WatchSource:0}: Error finding container a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db: Status 404 returned error can't find the container with id a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db Oct 30 01:28:38.071655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:38.071566 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" event={"ID":"2dab4abc-8b9f-4c49-b1d3-3c1593504be6","Type":"ContainerStarted","Data":"9eec52aa966fb5a3c41e31853187ef802fff574f97d3fafe2d0944f9a9a2cbb9"} Oct 30 01:28:38.071655 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:38.071622 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" event={"ID":"2dab4abc-8b9f-4c49-b1d3-3c1593504be6","Type":"ContainerStarted","Data":"a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db"} Oct 30 01:28:38.110608 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:38.110564 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" podStartSLOduration=2.110549687 podStartE2EDuration="2.110549687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:38.109617001 +0000 UTC m=+3370.349852140" watchObservedRunningTime="2026-10-30 01:28:38.110549687 +0000 UTC m=+3370.350784797" Oct 30 01:28:39.078984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:39.078950 5938 generic.go:356] "Generic (PLEG): container finished" podID="2dab4abc-8b9f-4c49-b1d3-3c1593504be6" containerID="9eec52aa966fb5a3c41e31853187ef802fff574f97d3fafe2d0944f9a9a2cbb9" exitCode=0 Oct 30 01:28:39.079389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:39.079041 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" event={"ID":"2dab4abc-8b9f-4c49-b1d3-3c1593504be6","Type":"ContainerDied","Data":"9eec52aa966fb5a3c41e31853187ef802fff574f97d3fafe2d0944f9a9a2cbb9"} Oct 30 01:28:40.224428 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.224404 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:40.327016 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.326988 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\") pod \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " Oct 30 01:28:40.327160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.327031 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\") pod \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " Oct 30 01:28:40.327160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.327062 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\" (UniqueName: \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\") pod \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\" (UID: \"2dab4abc-8b9f-4c49-b1d3-3c1593504be6\") " Oct 30 01:28:40.327331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.327309 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca" pod "2dab4abc-8b9f-4c49-b1d3-3c1593504be6" (UID: "2dab4abc-8b9f-4c49-b1d3-3c1593504be6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:40.329132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.329106 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca" pod "2dab4abc-8b9f-4c49-b1d3-3c1593504be6" (UID: "2dab4abc-8b9f-4c49-b1d3-3c1593504be6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:40.329240 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.329209 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4" pod "2dab4abc-8b9f-4c49-b1d3-3c1593504be6" (UID: "2dab4abc-8b9f-4c49-b1d3-3c1593504be6"). InnerVolumeSpecName "kube-api-access-x6qr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:40.427681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.427654 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:40.427681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.427681 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:40.427833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:40.427692 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6qr4\" (UniqueName: \"kubernetes.io/projected/2dab4abc-8b9f-4c49-b1d3-3c1593504be6-kube-api-access-x6qr4\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:28:41.091164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:41.091135 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" Oct 30 01:28:41.091164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:41.091155 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp" event={"ID":"2dab4abc-8b9f-4c49-b1d3-3c1593504be6","Type":"ContainerDied","Data":"a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db"} Oct 30 01:28:41.091383 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:41.091200 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b49d3b9626a408c306de75125f1d7fc11297854b3aa33c0d70d833b2e8e7db" Oct 30 01:28:45.136574 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:45.136528 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.310227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.310192 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp"] Oct 30 01:28:58.321746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.321718 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ttjqp"] Oct 30 01:28:58.354697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.354672 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb"] Oct 30 01:28:58.355101 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.355087 5938 cpu_manager.go:414] "containerMap: removing container" podUID="2dab4abc-8b9f-4c49-b1d3-3c1593504be6" containerName="revoke-keys" Oct 30 01:28:58.355166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.355105 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dab4abc-8b9f-4c49-b1d3-3c1593504be6" containerName="revoke-keys" Oct 30 01:28:58.355899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.355875 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dab4abc-8b9f-4c49-b1d3-3c1593504be6" containerName="revoke-keys" Oct 30 01:28:58.365427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.365402 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.378304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.378276 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb"] Oct 30 01:28:58.379556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.379529 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:28:58.379840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.379808 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:28:58.379966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.379818 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:28:58.379966 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.379899 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:28:58.421858 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.421835 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.421993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.421957 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqmw\" (UniqueName: \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.422052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.422039 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.523392 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.523367 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.523522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.523420 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqmw\" (UniqueName: \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.523522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.523482 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.524155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.524132 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.525551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.525531 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.545355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.545336 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqmw\" (UniqueName: \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\") pod \"maas-api-revoke-sub-3d4156f1-dkjbb\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.679205 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.679180 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:28:58.834074 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:28:58.834037 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:28:58.977502 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:58.977361 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb"] Oct 30 01:28:58.980030 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:28:58.980002 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3322e0_de10_4e70_933d_7fdacef82b23.slice/crio-04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41 WatchSource:0}: Error finding container 04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41: Status 404 returned error can't find the container with id 04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41 Oct 30 01:28:59.244433 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:59.244375 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" event={"ID":"4f3322e0-de10-4e70-933d-7fdacef82b23","Type":"ContainerStarted","Data":"04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41"} Oct 30 01:28:59.874891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:28:59.874851 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dab4abc-8b9f-4c49-b1d3-3c1593504be6" path="/var/lib/kubelet/pods/2dab4abc-8b9f-4c49-b1d3-3c1593504be6/volumes" Oct 30 01:29:00.252243 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:00.252218 5938 generic.go:356] "Generic (PLEG): container finished" podID="4f3322e0-de10-4e70-933d-7fdacef82b23" containerID="77f07f3a3b53fa39b232fe88f6ab212cc2e6ab157b8e2a23129cb58b122a32e2" exitCode=0 Oct 30 01:29:00.252394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:00.252262 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" event={"ID":"4f3322e0-de10-4e70-933d-7fdacef82b23","Type":"ContainerDied","Data":"77f07f3a3b53fa39b232fe88f6ab212cc2e6ab157b8e2a23129cb58b122a32e2"} Oct 30 01:29:01.392310 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.392282 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:29:01.451907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.451879 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\" (UniqueName: \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\") pod \"4f3322e0-de10-4e70-933d-7fdacef82b23\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " Oct 30 01:29:01.452060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.451948 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\") pod \"4f3322e0-de10-4e70-933d-7fdacef82b23\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " Oct 30 01:29:01.452129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.452070 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\") pod \"4f3322e0-de10-4e70-933d-7fdacef82b23\" (UID: \"4f3322e0-de10-4e70-933d-7fdacef82b23\") " Oct 30 01:29:01.452427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.452403 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca" pod "4f3322e0-de10-4e70-933d-7fdacef82b23" (UID: "4f3322e0-de10-4e70-933d-7fdacef82b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:01.454271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.454242 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw" pod "4f3322e0-de10-4e70-933d-7fdacef82b23" (UID: "4f3322e0-de10-4e70-933d-7fdacef82b23"). InnerVolumeSpecName "kube-api-access-2pqmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:01.454483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.454463 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca" pod "4f3322e0-de10-4e70-933d-7fdacef82b23" (UID: "4f3322e0-de10-4e70-933d-7fdacef82b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:01.552932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.552857 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:01.552932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.552901 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pqmw\" (UniqueName: \"kubernetes.io/projected/4f3322e0-de10-4e70-933d-7fdacef82b23-kube-api-access-2pqmw\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:01.552932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:01.552911 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3322e0-de10-4e70-933d-7fdacef82b23-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:02.265187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:02.265113 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" Oct 30 01:29:02.265187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:02.265124 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb" event={"ID":"4f3322e0-de10-4e70-933d-7fdacef82b23","Type":"ContainerDied","Data":"04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41"} Oct 30 01:29:02.265187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:02.265169 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04487a0dfdb67604621eff7427bf07544b98cd2860a4c7e8f26a52d5ba31fe41" Oct 30 01:29:06.027129 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:06.027090 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm"] Oct 30 01:29:06.038476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:06.038450 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d76e889-xc5qm"] Oct 30 01:29:07.874585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:07.874541 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a60250c-249e-4105-8828-d5b3e1d77628" path="/var/lib/kubelet/pods/3a60250c-249e-4105-8828-d5b3e1d77628/volumes" Oct 30 01:29:15.031168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:15.031135 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn"] Oct 30 01:29:15.042550 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:15.042522 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ae2d26-jm2xn"] Oct 30 01:29:15.874815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:15.874780 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f" path="/var/lib/kubelet/pods/fa2de4cb-0fd2-49f5-b59c-1e5f303cb70f/volumes" Oct 30 01:29:22.320853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.320817 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb"] Oct 30 01:29:22.332152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.332123 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dkjbb"] Oct 30 01:29:22.367955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.367925 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd"] Oct 30 01:29:22.368333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.368320 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4f3322e0-de10-4e70-933d-7fdacef82b23" containerName="revoke-keys" Oct 30 01:29:22.368384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.368335 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3322e0-de10-4e70-933d-7fdacef82b23" containerName="revoke-keys" Oct 30 01:29:22.368426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.368414 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3322e0-de10-4e70-933d-7fdacef82b23" containerName="revoke-keys" Oct 30 01:29:22.373035 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.373016 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.387204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.387155 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:29:22.387204 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.387202 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:29:22.387377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.387244 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:29:22.387377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.387244 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:29:22.389739 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.389708 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd"] Oct 30 01:29:22.523489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.523459 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.523628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.523529 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.523711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.523648 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7586\" (UniqueName: \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.624554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.624525 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.624675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.624608 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7586\" (UniqueName: \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.624675 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.624659 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.625265 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.625243 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.626891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.626852 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.646678 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.646655 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7586\" (UniqueName: \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\") pod \"maas-api-revoke-sub-3d4156f1-lggtd\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.693225 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.693190 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:22.986939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:22.986829 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd"] Oct 30 01:29:22.989769 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:29:22.989734 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6a68db_d7ef_43b8_bc6d_abfd2353fc95.slice/crio-3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced WatchSource:0}: Error finding container 3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced: Status 404 returned error can't find the container with id 3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced Oct 30 01:29:23.381377 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:23.381347 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" event={"ID":"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95","Type":"ContainerStarted","Data":"557313dc6f82234077cc46a85b338a73beb06410cf45ed46a011a54085917126"} Oct 30 01:29:23.381708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:23.381398 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" event={"ID":"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95","Type":"ContainerStarted","Data":"3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced"} Oct 30 01:29:23.420959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:23.420918 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" podStartSLOduration=1.420902492 podStartE2EDuration="1.420902492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:23.419785985 +0000 UTC m=+3415.660021099" watchObservedRunningTime="2026-10-30 01:29:23.420902492 +0000 UTC m=+3415.661137602" Oct 30 01:29:23.875357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:23.875329 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3322e0-de10-4e70-933d-7fdacef82b23" path="/var/lib/kubelet/pods/4f3322e0-de10-4e70-933d-7fdacef82b23/volumes" Oct 30 01:29:24.388935 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:24.388904 5938 generic.go:356] "Generic (PLEG): container finished" podID="dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" containerID="557313dc6f82234077cc46a85b338a73beb06410cf45ed46a011a54085917126" exitCode=0 Oct 30 01:29:24.389312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:24.388943 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" event={"ID":"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95","Type":"ContainerDied","Data":"557313dc6f82234077cc46a85b338a73beb06410cf45ed46a011a54085917126"} Oct 30 01:29:25.541793 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.541770 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:25.650353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.650292 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\") pod \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " Oct 30 01:29:25.650353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.650343 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\") pod \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " Oct 30 01:29:25.650566 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.650453 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\" (UniqueName: \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\") pod \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\" (UID: \"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95\") " Oct 30 01:29:25.650656 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.650637 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca" pod "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" (UID: "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:25.650834 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.650817 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.652466 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.652441 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586" pod "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" (UID: "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95"). InnerVolumeSpecName "kube-api-access-q7586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:25.652537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.652510 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca" pod "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" (UID: "dd6a68db-d7ef-43b8-bc6d-abfd2353fc95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:25.752104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.752083 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7586\" (UniqueName: \"kubernetes.io/projected/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-kube-api-access-q7586\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:25.752104 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:25.752102 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:29:26.401884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:26.401850 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" Oct 30 01:29:26.402011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:26.401897 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd" event={"ID":"dd6a68db-d7ef-43b8-bc6d-abfd2353fc95","Type":"ContainerDied","Data":"3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced"} Oct 30 01:29:26.402011 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:26.401946 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3514e38ecec1bae1760648436592609edb3b7fe2085d3cbd47b41b5c14506ced" Oct 30 01:29:28.834551 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:29:28.834511 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:29:32.397682 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.397650 5938 scope.go:122] "RemoveContainer" containerID="953199c0b005ddc4c9704591a7986eacfb6965898b37658c7759801d9cf90014" Oct 30 01:29:32.406849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.406828 5938 scope.go:122] "RemoveContainer" containerID="a1808fa8be50a357fa6b84feb5115569bdb259c79cfc65ca6975e6356d649ff7" Oct 30 01:29:32.417266 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.417242 5938 scope.go:122] "RemoveContainer" containerID="3dfa4b082b6e0e72d1d5775bd8da4d293871a6fb57cfb5a2050fe1dd056b1a32" Oct 30 01:29:32.425162 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.425140 5938 scope.go:122] "RemoveContainer" containerID="cd434a1120310d13926eaee15770f7c0beda2d9263b3da6ba31318778fef3e19" Oct 30 01:29:32.432653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.432629 5938 scope.go:122] "RemoveContainer" containerID="b5b2d8108c0925aca3cee342775e676460b2b0d91d67d1439745c2fa3a4e8972" Oct 30 01:29:32.440638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.440627 5938 scope.go:122] "RemoveContainer" containerID="7482387ff4866fad23510ba33f18c6e8913fdc3e26a92043890f8e591424a237" Oct 30 01:29:32.447901 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.447878 5938 scope.go:122] "RemoveContainer" containerID="8f11fdb4052cdc27cfa01c566d16d4c3fadd2788433fb5f773dc3c1888fe9420" Oct 30 01:29:32.455124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.455106 5938 scope.go:122] "RemoveContainer" containerID="990b80272e9f128a63d2d72b5026067e01788d85f5041441530874411cf3a83d" Oct 30 01:29:32.462122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.462109 5938 scope.go:122] "RemoveContainer" containerID="2a46e706d5d36c6d704f2b2875aa5f31f0b90f402809449b5b1d320d9223801f" Oct 30 01:29:32.469071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:32.469052 5938 scope.go:122] "RemoveContainer" containerID="704092c238d9ff6e492394df78342aa28cea992b4a5df2a20efb452e69ec8dd5" Oct 30 01:29:42.424330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:42.424242 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:43.909309 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:43.909271 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.384293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:44.384261 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.669519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:44.669444 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.817428 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:44.817397 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:47.490247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:47.490213 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:47.652787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:47.652755 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:47.809898 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:47.809804 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:49.247812 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:49.247735 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:49.383658 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:49.383618 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:51.103996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:51.103956 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:52.760968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:52.760933 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:54.766885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:54.766819 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:56.528365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:56.528324 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:58.456997 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:29:58.456963 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:58.833453 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:29:58.833375 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:30:00.146505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.146469 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:00.146946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.146888 5938 cpu_manager.go:414] "containerMap: removing container" podUID="dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" containerName="revoke-keys" Oct 30 01:30:00.146946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.146906 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" containerName="revoke-keys" Oct 30 01:30:00.147058 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.147019 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" containerName="revoke-keys" Oct 30 01:30:00.152581 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.152556 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.168382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.168355 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:00.177562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.177538 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:00.177727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.177542 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:00.177800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.177578 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:30:00.250478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.250452 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56f9\" (UniqueName: \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.250612 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.250513 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.290767 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.290742 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:00.351540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.351510 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.351725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.351587 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56f9\" (UniqueName: \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.352219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.352200 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.374072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.374046 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56f9\" (UniqueName: \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\") pod \"maas-api-key-cleanup-29888730-htcjs\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.466291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.466223 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:00.761246 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:00.761220 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:00.763825 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:30:00.763797 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0fa5f7_e29e_4935_ac44_bbb7d723d650.slice/crio-154bd14c59753bbacf9499b9bb2099e8a26c71e22f34dfcea9b423efcd263188 WatchSource:0}: Error finding container 154bd14c59753bbacf9499b9bb2099e8a26c71e22f34dfcea9b423efcd263188: Status 404 returned error can't find the container with id 154bd14c59753bbacf9499b9bb2099e8a26c71e22f34dfcea9b423efcd263188 Oct 30 01:30:01.593241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:01.593202 5938 generic.go:356] "Generic (PLEG): container finished" podID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerID="412664169e870c3ef8617baafc59f1215f171af1e2f7e3e1bd1e75d21c28b138" exitCode=60 Oct 30 01:30:01.593676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:01.593284 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" event={"ID":"3c0fa5f7-e29e-4935-ac44-bbb7d723d650","Type":"ContainerDied","Data":"412664169e870c3ef8617baafc59f1215f171af1e2f7e3e1bd1e75d21c28b138"} Oct 30 01:30:01.593676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:01.593361 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" event={"ID":"3c0fa5f7-e29e-4935-ac44-bbb7d723d650","Type":"ContainerStarted","Data":"154bd14c59753bbacf9499b9bb2099e8a26c71e22f34dfcea9b423efcd263188"} Oct 30 01:30:01.593676 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:01.593573 5938 scope.go:122] "RemoveContainer" containerID="412664169e870c3ef8617baafc59f1215f171af1e2f7e3e1bd1e75d21c28b138" Oct 30 01:30:01.922286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:01.922249 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:02.474563 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:02.474531 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:02.600534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:02.600495 5938 generic.go:356] "Generic (PLEG): container finished" podID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerID="86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f" exitCode=60 Oct 30 01:30:02.601050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:02.600582 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" event={"ID":"3c0fa5f7-e29e-4935-ac44-bbb7d723d650","Type":"ContainerDied","Data":"86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f"} Oct 30 01:30:02.601050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:02.600658 5938 scope.go:122] "RemoveContainer" containerID="412664169e870c3ef8617baafc59f1215f171af1e2f7e3e1bd1e75d21c28b138" Oct 30 01:30:02.601050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:02.600970 5938 scope.go:122] "RemoveContainer" containerID="86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f" Oct 30 01:30:02.601248 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:30:02.601228 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-htcjs_odh-ai-gateway-infra(3c0fa5f7-e29e-4935-ac44-bbb7d723d650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" Oct 30 01:30:03.072505 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:03.072461 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.219133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:03.219080 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.364974 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:03.364943 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.607398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:03.607371 5938 scope.go:122] "RemoveContainer" containerID="86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f" Oct 30 01:30:03.607806 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:30:03.607587 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-htcjs_odh-ai-gateway-infra(3c0fa5f7-e29e-4935-ac44-bbb7d723d650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" Oct 30 01:30:03.746543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:03.746462 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:04.036696 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:04.036595 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt"] Oct 30 01:30:04.040459 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:04.040427 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rqtbt"] Oct 30 01:30:04.973844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:04.973814 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:05.500562 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:05.500520 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:05.874959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:05.874930 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600f0507-2dcc-44e3-b07f-4d77b3c9e86f" path="/var/lib/kubelet/pods/600f0507-2dcc-44e3-b07f-4d77b3c9e86f/volumes" Oct 30 01:30:07.083951 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:07.083904 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:07.358855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:07.358825 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:09.107648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:09.107609 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:10.897918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:10.897886 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:12.532071 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:12.532035 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:14.374023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:14.373990 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:14.868883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:14.868842 5938 scope.go:122] "RemoveContainer" containerID="86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f" Oct 30 01:30:15.669660 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:15.669632 5938 generic.go:356] "Generic (PLEG): container finished" podID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerID="a5d8ca5338e78166673252ab59b05f19f1e4a553119d823ce41bb876bd61cb5d" exitCode=60 Oct 30 01:30:15.670030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:15.669701 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" event={"ID":"3c0fa5f7-e29e-4935-ac44-bbb7d723d650","Type":"ContainerDied","Data":"a5d8ca5338e78166673252ab59b05f19f1e4a553119d823ce41bb876bd61cb5d"} Oct 30 01:30:15.670030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:15.669765 5938 scope.go:122] "RemoveContainer" containerID="86c14fa99df9107b52a34a931eb19a5de7b873fed704f3e11f04c6748ebdec5f" Oct 30 01:30:15.670030 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:15.670022 5938 scope.go:122] "RemoveContainer" containerID="a5d8ca5338e78166673252ab59b05f19f1e4a553119d823ce41bb876bd61cb5d" Oct 30 01:30:15.670263 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:30:15.670236 5938 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-htcjs_odh-ai-gateway-infra(3c0fa5f7-e29e-4935-ac44-bbb7d723d650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" Oct 30 01:30:16.041888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:16.041838 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:16.717763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:16.717731 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:16.853177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:16.853154 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:17.005883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.005792 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\") pod \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " Oct 30 01:30:17.005883 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.005853 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\" (UniqueName: \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\") pod \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\" (UID: \"3c0fa5f7-e29e-4935-ac44-bbb7d723d650\") " Oct 30 01:30:17.006235 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.006211 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca" pod "3c0fa5f7-e29e-4935-ac44-bbb7d723d650" (UID: "3c0fa5f7-e29e-4935-ac44-bbb7d723d650"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:17.007856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.007832 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9" pod "3c0fa5f7-e29e-4935-ac44-bbb7d723d650" (UID: "3c0fa5f7-e29e-4935-ac44-bbb7d723d650"). InnerVolumeSpecName "kube-api-access-m56f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:17.107188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.107150 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-maas-api-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:17.107188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.107181 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56f9\" (UniqueName: \"kubernetes.io/projected/3c0fa5f7-e29e-4935-ac44-bbb7d723d650-kube-api-access-m56f9\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:17.682346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.682307 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" Oct 30 01:30:17.682514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.682341 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs" event={"ID":"3c0fa5f7-e29e-4935-ac44-bbb7d723d650","Type":"ContainerDied","Data":"154bd14c59753bbacf9499b9bb2099e8a26c71e22f34dfcea9b423efcd263188"} Oct 30 01:30:17.682514 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.682414 5938 scope.go:122] "RemoveContainer" containerID="a5d8ca5338e78166673252ab59b05f19f1e4a553119d823ce41bb876bd61cb5d" Oct 30 01:30:17.730145 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.730118 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:17.741395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.741374 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-htcjs"] Oct 30 01:30:17.745830 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.745807 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:17.875184 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:17.875151 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" path="/var/lib/kubelet/pods/3c0fa5f7-e29e-4935-ac44-bbb7d723d650/volumes" Oct 30 01:30:19.492079 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:19.492045 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:21.370587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:21.369476 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:23.263323 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:23.263288 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:24.949379 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:24.949340 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:26.551890 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:26.551838 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:28.102143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102110 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z"] Oct 30 01:30:28.102559 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102544 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.102623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102560 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.102623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102584 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.102623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102589 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.102724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102658 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.102724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.102671 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:28.107478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.107452 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.121692 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.121671 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:28.122192 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.122168 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:28.133420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.133397 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:28.133708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.133692 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:30:28.136312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.136288 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z"] Oct 30 01:30:28.198826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.198802 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnzb\" (UniqueName: \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.198969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.198846 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.198969 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.198949 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.297848 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.297818 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:28.299948 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.299929 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.300031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.299966 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.300094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.300033 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnzb\" (UniqueName: \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.300672 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.300649 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.302167 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.302147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.323631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.323597 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnzb\" (UniqueName: \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\") pod \"maas-api-revoke-sub-5a417f94-cfm8z\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.421727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.421657 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:28.716330 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.716304 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z"] Oct 30 01:30:28.719196 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:30:28.719168 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode991ab6e_9ef0_423c_b530_d727fad65144.slice/crio-6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0 WatchSource:0}: Error finding container 6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0: Status 404 returned error can't find the container with id 6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0 Oct 30 01:30:28.739315 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:28.739286 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" event={"ID":"e991ab6e-9ef0-423c-b530-d727fad65144","Type":"ContainerStarted","Data":"6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0"} Oct 30 01:30:28.833449 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:30:28.833420 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:30:29.747669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:29.747599 5938 generic.go:356] "Generic (PLEG): container finished" podID="e991ab6e-9ef0-423c-b530-d727fad65144" containerID="7bbbfc5ed31a27928d76548ea8ad37c1911751048b219b9430d11eb19590b1d2" exitCode=0 Oct 30 01:30:29.748080 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:29.747692 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" event={"ID":"e991ab6e-9ef0-423c-b530-d727fad65144","Type":"ContainerDied","Data":"7bbbfc5ed31a27928d76548ea8ad37c1911751048b219b9430d11eb19590b1d2"} Oct 30 01:30:29.992410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:29.992380 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:30.889594 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:30.889566 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:31.022987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.022917 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\" (UniqueName: \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\") pod \"e991ab6e-9ef0-423c-b530-d727fad65144\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " Oct 30 01:30:31.022987 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.022985 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\") pod \"e991ab6e-9ef0-423c-b530-d727fad65144\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " Oct 30 01:30:31.023199 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.023041 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\") pod \"e991ab6e-9ef0-423c-b530-d727fad65144\" (UID: \"e991ab6e-9ef0-423c-b530-d727fad65144\") " Oct 30 01:30:31.023447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.023410 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca" pod "e991ab6e-9ef0-423c-b530-d727fad65144" (UID: "e991ab6e-9ef0-423c-b530-d727fad65144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:31.025004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.024980 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca" pod "e991ab6e-9ef0-423c-b530-d727fad65144" (UID: "e991ab6e-9ef0-423c-b530-d727fad65144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:31.025103 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.025084 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb" pod "e991ab6e-9ef0-423c-b530-d727fad65144" (UID: "e991ab6e-9ef0-423c-b530-d727fad65144"). InnerVolumeSpecName "kube-api-access-mtnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:31.124695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.124665 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:31.124695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.124691 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e991ab6e-9ef0-423c-b530-d727fad65144-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:31.124846 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.124702 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtnzb\" (UniqueName: \"kubernetes.io/projected/e991ab6e-9ef0-423c-b530-d727fad65144-kube-api-access-mtnzb\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:31.761356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.761325 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" event={"ID":"e991ab6e-9ef0-423c-b530-d727fad65144","Type":"ContainerDied","Data":"6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0"} Oct 30 01:30:31.761525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.761382 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6539ebbb63aac597ee8a3218bdd9ef2f66ee8409aa90c9bf4f9424ecdc0cb4b0" Oct 30 01:30:31.761525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.761335 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z" Oct 30 01:30:31.893014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:31.892989 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.522244 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:32.522210 5938 scope.go:122] "RemoveContainer" containerID="0de65f77e4e5b48fd9a53df69194fa259eab7146836255a14fb618b1614efdf0" Oct 30 01:30:32.531040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:32.531014 5938 scope.go:122] "RemoveContainer" containerID="d3564ac978b4fa35520daec865648b64c6ebfe5bb75be77c74a021ccfd98f923" Oct 30 01:30:32.539049 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:32.539027 5938 scope.go:122] "RemoveContainer" containerID="2021c94a81954e77956d868663a8019952ad1346c04cc16ff8bfcc52ac6f1e09" Oct 30 01:30:32.546694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:32.546676 5938 scope.go:122] "RemoveContainer" containerID="c252cc5370e761be20e505bcfbffa54a908896908cc6d6d0f50b53f0c23671cd" Oct 30 01:30:32.554460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:32.554440 5938 scope.go:122] "RemoveContainer" containerID="28a7cd1201ee7c4e276a6d9c606ec121f367747d14046088a8bb1e1de173c920" Oct 30 01:30:33.548686 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:33.548649 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:35.233929 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:35.233891 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:36.860414 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:36.860377 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:38.495498 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:38.495464 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.211217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.211178 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.353959 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.353929 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv"] Oct 30 01:30:39.354299 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354285 5938 cpu_manager.go:414] "containerMap: removing container" podUID="e991ab6e-9ef0-423c-b530-d727fad65144" containerName="revoke-keys" Oct 30 01:30:39.354352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354300 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991ab6e-9ef0-423c-b530-d727fad65144" containerName="revoke-keys" Oct 30 01:30:39.354352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354325 5938 cpu_manager.go:414] "containerMap: removing container" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:39.354352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354330 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:39.354527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354387 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0fa5f7-e29e-4935-ac44-bbb7d723d650" containerName="cleanup" Oct 30 01:30:39.354527 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.354399 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991ab6e-9ef0-423c-b530-d727fad65144" containerName="revoke-keys" Oct 30 01:30:39.359595 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.359578 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.375312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.375291 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv"] Oct 30 01:30:39.384800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.384776 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:39.384800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.384795 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:39.385029 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.384794 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:39.385147 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.385121 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:30:39.400714 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.400691 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9nk8\" (UniqueName: \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.400815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.400730 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.400930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.400907 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.501736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.501675 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.501736 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.501713 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9nk8\" (UniqueName: \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.502208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.501758 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.502431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.502409 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.504178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.504154 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.523937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.523916 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9nk8\" (UniqueName: \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.671881 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.671827 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:39.966960 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:39.966935 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv"] Oct 30 01:30:39.969560 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:30:39.969527 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af3d97b_ce0f_4b93_9121_8cad4f3ad5d3.slice/crio-1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888 WatchSource:0}: Error finding container 1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888: Status 404 returned error can't find the container with id 1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888 Oct 30 01:30:40.371743 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:40.371695 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:40.810934 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:40.810903 5938 generic.go:356] "Generic (PLEG): container finished" podID="4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" containerID="6a4f48a885d9a6dd4fc011d7c4544352a0f7b866578788ee9bada71992a77532" exitCode=0 Oct 30 01:30:40.811279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:40.810983 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" event={"ID":"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3","Type":"ContainerDied","Data":"6a4f48a885d9a6dd4fc011d7c4544352a0f7b866578788ee9bada71992a77532"} Oct 30 01:30:40.811279 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:40.811029 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" event={"ID":"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3","Type":"ContainerStarted","Data":"1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888"} Oct 30 01:30:41.943423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:41.943398 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:42.021124 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.021100 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\" (UniqueName: \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\") pod \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " Oct 30 01:30:42.021267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.021178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\") pod \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " Oct 30 01:30:42.021267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.021207 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\") pod \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\" (UID: \"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3\") " Oct 30 01:30:42.021625 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.021599 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca" pod "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" (UID: "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:42.023067 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.023043 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8" pod "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" (UID: "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3"). InnerVolumeSpecName "kube-api-access-m9nk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:42.023172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.023150 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca" pod "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" (UID: "4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:42.122553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.122525 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.122553 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.122553 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.122708 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.122563 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9nk8\" (UniqueName: \"kubernetes.io/projected/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3-kube-api-access-m9nk8\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:30:42.400222 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.400131 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.603654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.603626 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.823166 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.823098 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" Oct 30 01:30:42.823312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.823101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv" event={"ID":"4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3","Type":"ContainerDied","Data":"1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888"} Oct 30 01:30:42.823312 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:42.823240 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f77ef65c519e2f452f0e1b56bac26414b0e854f32fe3cc19aa38f5d0f6dd888" Oct 30 01:30:43.954643 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:43.954614 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.268716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:44.268640 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:45.641003 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:45.640944 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:47.387428 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:47.387382 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:49.124368 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:49.124245 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:49.935614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:49.935577 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:50.076638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:50.076608 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:50.912362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:50.912307 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.405173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:52.405128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.915932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:52.915892 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.987962 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:52.987933 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv"] Oct 30 01:30:52.999424 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:52.999399 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3ab7f1-6hnbv"] Oct 30 01:30:53.082234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:53.082183 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:53.537516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:53.537480 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:53.874921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:53.874888 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" path="/var/lib/kubelet/pods/4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3/volumes" Oct 30 01:30:55.025126 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:55.025089 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn"] Oct 30 01:30:55.036455 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:55.036424 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hvfpn"] Oct 30 01:30:55.875411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:55.875377 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a354e2-6547-41bb-b06c-08f8a0dbea25" path="/var/lib/kubelet/pods/60a354e2-6547-41bb-b06c-08f8a0dbea25/volumes" Oct 30 01:30:58.834113 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:30:58.834070 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:30:59.356461 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:59.356430 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:59.495847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:59.495809 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:59.754217 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:30:59.754138 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:01.006458 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.006408 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd"] Oct 30 01:31:01.017713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.017683 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lggtd"] Oct 30 01:31:01.061522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.061498 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk"] Oct 30 01:31:01.061907 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.061893 5938 cpu_manager.go:414] "containerMap: removing container" podUID="4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" containerName="revoke-keys" Oct 30 01:31:01.061967 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.061909 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" containerName="revoke-keys" Oct 30 01:31:01.062006 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.061996 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af3d97b-ce0f-4b93-9121-8cad4f3ad5d3" containerName="revoke-keys" Oct 30 01:31:01.066155 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.066140 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.081426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.081405 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:01.081649 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.081631 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:31:01.081741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.081583 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:01.081741 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.081556 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:01.083353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.083332 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk"] Oct 30 01:31:01.187674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.187644 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.187815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.187729 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmkc\" (UniqueName: \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.187815 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.187767 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.289146 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.289086 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmkc\" (UniqueName: \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.289252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.289145 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.289252 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.289212 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.289816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.289790 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.291453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.291431 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.311301 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.311279 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmkc\" (UniqueName: \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\") pod \"maas-api-revoke-sub-3d4156f1-r5wrk\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.378213 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.378193 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:01.673314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.673258 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk"] Oct 30 01:31:01.676016 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:31:01.675988 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37907747_6761_44a7_9710_509e69fe0e04.slice/crio-a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12 WatchSource:0}: Error finding container a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12: Status 404 returned error can't find the container with id a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12 Oct 30 01:31:01.874547 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.874520 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6a68db-d7ef-43b8-bc6d-abfd2353fc95" path="/var/lib/kubelet/pods/dd6a68db-d7ef-43b8-bc6d-abfd2353fc95/volumes" Oct 30 01:31:01.917894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:01.917853 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" event={"ID":"37907747-6761-44a7-9710-509e69fe0e04","Type":"ContainerStarted","Data":"a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12"} Oct 30 01:31:02.924667 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:02.924636 5938 generic.go:356] "Generic (PLEG): container finished" podID="37907747-6761-44a7-9710-509e69fe0e04" containerID="37b4be80c0dccf31e36b2b330175cf6fe6fb20754b839326fd60d9e4d5a83c16" exitCode=0 Oct 30 01:31:02.925068 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:02.924715 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" event={"ID":"37907747-6761-44a7-9710-509e69fe0e04","Type":"ContainerDied","Data":"37b4be80c0dccf31e36b2b330175cf6fe6fb20754b839326fd60d9e4d5a83c16"} Oct 30 01:31:04.057668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.057644 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:04.111662 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.111637 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\") pod \"37907747-6761-44a7-9710-509e69fe0e04\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " Oct 30 01:31:04.111820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.111699 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\") pod \"37907747-6761-44a7-9710-509e69fe0e04\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " Oct 30 01:31:04.111820 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.111754 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\" (UniqueName: \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\") pod \"37907747-6761-44a7-9710-509e69fe0e04\" (UID: \"37907747-6761-44a7-9710-509e69fe0e04\") " Oct 30 01:31:04.112143 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.112115 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca" pod "37907747-6761-44a7-9710-509e69fe0e04" (UID: "37907747-6761-44a7-9710-509e69fe0e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:04.113795 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.113771 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca" pod "37907747-6761-44a7-9710-509e69fe0e04" (UID: "37907747-6761-44a7-9710-509e69fe0e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:04.113900 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.113793 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc" pod "37907747-6761-44a7-9710-509e69fe0e04" (UID: "37907747-6761-44a7-9710-509e69fe0e04"). InnerVolumeSpecName "kube-api-access-cnmkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:04.213257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.213194 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37907747-6761-44a7-9710-509e69fe0e04-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:31:04.213257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.213222 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37907747-6761-44a7-9710-509e69fe0e04-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:31:04.213257 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.213238 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnmkc\" (UniqueName: \"kubernetes.io/projected/37907747-6761-44a7-9710-509e69fe0e04-kube-api-access-cnmkc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:31:04.935350 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.935319 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" Oct 30 01:31:04.935529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.935354 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk" event={"ID":"37907747-6761-44a7-9710-509e69fe0e04","Type":"ContainerDied","Data":"a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12"} Oct 30 01:31:04.935529 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:04.935398 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2b01bf65f938d5d6fa654a0b226a98a56c9eabbb10d91539483cfdda3034a12" Oct 30 01:31:15.044355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:15.043964 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:31:15.044355 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:15.044041 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:31:15.332786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:15.332184 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:31:15.332786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:15.332255 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:31:19.026090 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:19.026055 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp"] Oct 30 01:31:19.038208 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:19.038177 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bc8rp"] Oct 30 01:31:19.874467 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:19.874436 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fbc932-37fc-44cf-b8b9-af36dbc44873" path="/var/lib/kubelet/pods/c8fbc932-37fc-44cf-b8b9-af36dbc44873/volumes" Oct 30 01:31:21.475452 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:21.475401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:21.751371 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:21.751286 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:27.499337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:27.499302 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:28.842307 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:31:28.842254 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:31:30.036771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:30.036736 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6"] Oct 30 01:31:30.048056 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:30.048027 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-rf6d6"] Oct 30 01:31:31.876446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:31.876413 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f73258-6ff6-4b8a-9177-bc5ec5c795b4" path="/var/lib/kubelet/pods/b8f73258-6ff6-4b8a-9177-bc5ec5c795b4/volumes" Oct 30 01:31:32.591434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:32.591403 5938 scope.go:122] "RemoveContainer" containerID="f17fc245aec81b0495f85bff41c819a481c516b43a5dcdd6a27e3231b0c294d2" Oct 30 01:31:42.720761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:42.720721 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.026545 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:57.026504 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r"] Oct 30 01:31:57.037526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:57.037495 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-m2v8r"] Oct 30 01:31:57.874589 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:31:57.874555 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355fae7e-f48f-42c7-bc89-312fa052e561" path="/var/lib/kubelet/pods/355fae7e-f48f-42c7-bc89-312fa052e561/volumes" Oct 30 01:31:58.834113 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:31:58.834077 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:32:08.036629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:08.036598 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj"] Oct 30 01:32:08.038239 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:08.038211 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5qbmj"] Oct 30 01:32:09.874847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:09.874788 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfb8496-37d7-4e03-8757-70001bfce1bf" path="/var/lib/kubelet/pods/0bfb8496-37d7-4e03-8757-70001bfce1bf/volumes" Oct 30 01:32:28.833685 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:32:28.833649 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:32:30.044911 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:30.044851 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl"] Oct 30 01:32:30.056270 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:30.056233 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-24bpl"] Oct 30 01:32:30.731116 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:30.731089 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:32:30.772847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:30.772825 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:32:31.874002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:31.873972 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954d7640-4f34-44f2-8e6e-d11574988cf5" path="/var/lib/kubelet/pods/954d7640-4f34-44f2-8e6e-d11574988cf5/volumes" Oct 30 01:32:32.613651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:32.613614 5938 scope.go:122] "RemoveContainer" containerID="035f02029c7dd640e6e179f21bd369cb642d82242f6e6ed46b61e3113cb50c16" Oct 30 01:32:32.621859 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:32.621841 5938 scope.go:122] "RemoveContainer" containerID="0b55d321c62d735210bbe85975b03e1ce556edeec4e28243c59c93c50371cb3c" Oct 30 01:32:32.629980 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:32.629960 5938 scope.go:122] "RemoveContainer" containerID="ec15d2317497504530052a298722bd0fb4e22408c87a6a8b8aeaca3138e98444" Oct 30 01:32:40.565220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:40.565130 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:42.857981 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:42.857943 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:52.033410 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:52.033370 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g"] Oct 30 01:32:52.044697 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:52.044667 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-n8f2g"] Oct 30 01:32:53.890616 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:53.890578 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a61fe7-7fff-4d4c-858a-977d28e7dcf1" path="/var/lib/kubelet/pods/36a61fe7-7fff-4d4c-858a-977d28e7dcf1/volumes" Oct 30 01:32:55.820064 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.820025 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz"] Oct 30 01:32:55.820468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.820454 5938 cpu_manager.go:414] "containerMap: removing container" podUID="37907747-6761-44a7-9710-509e69fe0e04" containerName="revoke-keys" Oct 30 01:32:55.820526 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.820469 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37907747-6761-44a7-9710-509e69fe0e04" containerName="revoke-keys" Oct 30 01:32:55.820570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.820559 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="37907747-6761-44a7-9710-509e69fe0e04" containerName="revoke-keys" Oct 30 01:32:55.825288 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.825269 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.839932 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.839905 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:55.851543 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.851519 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:55.851687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.851569 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:55.851687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.851607 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:32:55.853314 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.853293 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz"] Oct 30 01:32:55.897382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.897357 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.897483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.897402 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2brd\" (UniqueName: \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.897532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.897483 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.998751 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.998724 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.998853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.998769 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2brd\" (UniqueName: \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.998853 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.998805 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:55.999434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:55.999415 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:56.001173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:56.001147 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:56.021522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:56.021500 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2brd\" (UniqueName: \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\") pod \"maas-api-revoke-sub-d576e68b-mq5dz\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:56.137724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:56.137702 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:56.431777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:56.431679 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz"] Oct 30 01:32:56.434763 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:32:56.434735 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b76a46_a496_460c_be8e_0147ba6cfe01.slice/crio-bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce WatchSource:0}: Error finding container bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce: Status 404 returned error can't find the container with id bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce Oct 30 01:32:56.576479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:56.576451 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" event={"ID":"11b76a46-a496-460c-be8e-0147ba6cfe01","Type":"ContainerStarted","Data":"bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce"} Oct 30 01:32:57.582688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:57.582620 5938 generic.go:356] "Generic (PLEG): container finished" podID="11b76a46-a496-460c-be8e-0147ba6cfe01" containerID="c9769c7a3075baf3b01c9a27822a03636aa3770fc5a2ccfef3654f7fbe751323" exitCode=0 Oct 30 01:32:57.583043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:57.582703 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" event={"ID":"11b76a46-a496-460c-be8e-0147ba6cfe01","Type":"ContainerDied","Data":"c9769c7a3075baf3b01c9a27822a03636aa3770fc5a2ccfef3654f7fbe751323"} Oct 30 01:32:58.721444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.721418 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:58.823119 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.823081 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\") pod \"11b76a46-a496-460c-be8e-0147ba6cfe01\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " Oct 30 01:32:58.823308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.823217 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\" (UniqueName: \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\") pod \"11b76a46-a496-460c-be8e-0147ba6cfe01\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " Oct 30 01:32:58.823308 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.823292 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\") pod \"11b76a46-a496-460c-be8e-0147ba6cfe01\" (UID: \"11b76a46-a496-460c-be8e-0147ba6cfe01\") " Oct 30 01:32:58.823457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.823430 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca" pod "11b76a46-a496-460c-be8e-0147ba6cfe01" (UID: "11b76a46-a496-460c-be8e-0147ba6cfe01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:58.823665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.823637 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:32:58.825267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.825243 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca" pod "11b76a46-a496-460c-be8e-0147ba6cfe01" (UID: "11b76a46-a496-460c-be8e-0147ba6cfe01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:58.825394 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.825275 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd" pod "11b76a46-a496-460c-be8e-0147ba6cfe01" (UID: "11b76a46-a496-460c-be8e-0147ba6cfe01"). InnerVolumeSpecName "kube-api-access-m2brd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:58.833314 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:32:58.833283 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:32:58.924810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.924787 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2brd\" (UniqueName: \"kubernetes.io/projected/11b76a46-a496-460c-be8e-0147ba6cfe01-kube-api-access-m2brd\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:32:58.924810 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:58.924810 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b76a46-a496-460c-be8e-0147ba6cfe01-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:32:59.594738 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:59.594712 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" Oct 30 01:32:59.594920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:59.594712 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz" event={"ID":"11b76a46-a496-460c-be8e-0147ba6cfe01","Type":"ContainerDied","Data":"bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce"} Oct 30 01:32:59.594977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:32:59.594916 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae7b565c950f2fec6da23f286a25a0715e10320c6411d991fc8f26feb5dbfce" Oct 30 01:33:12.677164 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.677128 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc"] Oct 30 01:33:12.677647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.677553 5938 cpu_manager.go:414] "containerMap: removing container" podUID="11b76a46-a496-460c-be8e-0147ba6cfe01" containerName="revoke-keys" Oct 30 01:33:12.677647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.677564 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b76a46-a496-460c-be8e-0147ba6cfe01" containerName="revoke-keys" Oct 30 01:33:12.677647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.677630 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b76a46-a496-460c-be8e-0147ba6cfe01" containerName="revoke-keys" Oct 30 01:33:12.680638 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.680614 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.697775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.697755 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:33:12.709645 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.709622 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:33:12.710307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.710273 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:33:12.710447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.710354 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:33:12.712653 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.712631 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc"] Oct 30 01:33:12.766170 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.766142 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.766304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.766190 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsdlc\" (UniqueName: \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.766352 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.766303 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.867681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.867651 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.867826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.867695 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdlc\" (UniqueName: \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.867826 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.867743 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.868337 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.868320 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.870026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.870001 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.890945 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.890922 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdlc\" (UniqueName: \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\") pod \"maas-api-revoke-sub-d2bc93bc-ls2fc\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:12.993495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:12.993440 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:13.029105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.029077 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg"] Oct 30 01:33:13.034931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.034906 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-x22kg"] Oct 30 01:33:13.294765 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.294702 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc"] Oct 30 01:33:13.301212 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:33:13.300417 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31b2986_29c6_49a2_95db_40e51c5ce410.slice/crio-fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921 WatchSource:0}: Error finding container fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921: Status 404 returned error can't find the container with id fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921 Oct 30 01:33:13.688839 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.688803 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" event={"ID":"a31b2986-29c6-49a2-95db-40e51c5ce410","Type":"ContainerStarted","Data":"498e0452bb836cde26ed3324eecdb3f5a6a30da90696b57531b67af803c7f905"} Oct 30 01:33:13.689268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.688854 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" event={"ID":"a31b2986-29c6-49a2-95db-40e51c5ce410","Type":"ContainerStarted","Data":"fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921"} Oct 30 01:33:13.736687 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.736642 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" podStartSLOduration=1.73663023 podStartE2EDuration="1.73663023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:13.736403276 +0000 UTC m=+3645.976638390" watchObservedRunningTime="2026-10-30 01:33:13.73663023 +0000 UTC m=+3645.976865342" Oct 30 01:33:13.874450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:13.874415 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790e2615-f73c-4564-9d90-f8df4ffaaa73" path="/var/lib/kubelet/pods/790e2615-f73c-4564-9d90-f8df4ffaaa73/volumes" Oct 30 01:33:14.695996 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:14.695957 5938 generic.go:356] "Generic (PLEG): container finished" podID="a31b2986-29c6-49a2-95db-40e51c5ce410" containerID="498e0452bb836cde26ed3324eecdb3f5a6a30da90696b57531b67af803c7f905" exitCode=0 Oct 30 01:33:14.696411 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:14.696040 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" event={"ID":"a31b2986-29c6-49a2-95db-40e51c5ce410","Type":"ContainerDied","Data":"498e0452bb836cde26ed3324eecdb3f5a6a30da90696b57531b67af803c7f905"} Oct 30 01:33:15.825013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.824989 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:15.995621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.995559 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\") pod \"a31b2986-29c6-49a2-95db-40e51c5ce410\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " Oct 30 01:33:15.995766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.995640 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\" (UniqueName: \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\") pod \"a31b2986-29c6-49a2-95db-40e51c5ce410\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " Oct 30 01:33:15.995766 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.995676 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\") pod \"a31b2986-29c6-49a2-95db-40e51c5ce410\" (UID: \"a31b2986-29c6-49a2-95db-40e51c5ce410\") " Oct 30 01:33:15.995963 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.995939 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca" pod "a31b2986-29c6-49a2-95db-40e51c5ce410" (UID: "a31b2986-29c6-49a2-95db-40e51c5ce410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:15.997819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.997794 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca" pod "a31b2986-29c6-49a2-95db-40e51c5ce410" (UID: "a31b2986-29c6-49a2-95db-40e51c5ce410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:15.997971 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:15.997817 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc" pod "a31b2986-29c6-49a2-95db-40e51c5ce410" (UID: "a31b2986-29c6-49a2-95db-40e51c5ce410"). InnerVolumeSpecName "kube-api-access-wsdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:16.096740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.096710 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsdlc\" (UniqueName: \"kubernetes.io/projected/a31b2986-29c6-49a2-95db-40e51c5ce410-kube-api-access-wsdlc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.096740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.096737 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.096933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.096751 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31b2986-29c6-49a2-95db-40e51c5ce410-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:33:16.708304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.708273 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" Oct 30 01:33:16.708304 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.708288 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc" event={"ID":"a31b2986-29c6-49a2-95db-40e51c5ce410","Type":"ContainerDied","Data":"fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921"} Oct 30 01:33:16.708544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:16.708334 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb613738fc0f0343f947dea30fa0d0aa14beb5dddd6b826cc06f17b3545b3921" Oct 30 01:33:24.388396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.388329 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:24.750641 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.750551 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:33:24.751234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.751213 5938 cpu_manager.go:414] "containerMap: removing container" podUID="a31b2986-29c6-49a2-95db-40e51c5ce410" containerName="revoke-keys" Oct 30 01:33:24.751234 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.751234 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31b2986-29c6-49a2-95db-40e51c5ce410" containerName="revoke-keys" Oct 30 01:33:24.751431 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.751316 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31b2986-29c6-49a2-95db-40e51c5ce410" containerName="revoke-keys" Oct 30 01:33:24.754914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.754883 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.769576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.769554 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-fe8356-openshift-default-dockercfg-jc424\"" Oct 30 01:33:24.778587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778564 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.778695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778612 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.778757 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778718 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.778817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778760 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.778938 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778916 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.779212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.778980 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.779212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.779084 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.779212 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.779130 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wdc\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.779364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.779215 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.784276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.784251 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:33:24.830542 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.830517 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:24.880489 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880465 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880497 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880524 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880591 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880546 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880666 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880779 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880737 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880794 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880824 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wdc\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.880937 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880887 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.881115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880976 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.881115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.880998 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.881247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.881206 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.881317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.881287 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.881421 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.881398 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.882892 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.882858 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.883133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.883115 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.903238 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.903212 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:24.903348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:24.903244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wdc\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\") pod \"e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:25.030847 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.030768 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:25.068322 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.068294 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:25.381038 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.380969 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:33:25.383340 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:33:25.383313 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4567cd8_e736_4cfe_96ca_ecac7d69b8e2.slice/crio-8779dc04e93cb4459c9a19d69354830a00b9e8f832df68b0c6ffc87f9d7c706a WatchSource:0}: Error finding container 8779dc04e93cb4459c9a19d69354830a00b9e8f832df68b0c6ffc87f9d7c706a: Status 404 returned error can't find the container with id 8779dc04e93cb4459c9a19d69354830a00b9e8f832df68b0c6ffc87f9d7c706a Oct 30 01:33:25.385447 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.385414 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:25.385522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.385479 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:25.385522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.385508 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:25.755712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.755619 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerStarted","Data":"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428"} Oct 30 01:33:25.755712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.755677 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerStarted","Data":"8779dc04e93cb4459c9a19d69354830a00b9e8f832df68b0c6ffc87f9d7c706a"} Oct 30 01:33:25.800851 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:25.800806 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podStartSLOduration=1.800791847 podStartE2EDuration="1.800791847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:25.799219659 +0000 UTC m=+3658.039454771" watchObservedRunningTime="2026-10-30 01:33:25.800791847 +0000 UTC m=+3658.041026962" Oct 30 01:33:26.068816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:26.068736 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:26.073360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:26.073335 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:26.761019 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:26.760991 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:26.762086 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:26.762062 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:28.291630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:28.291590 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:28.472026 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:28.471993 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:28.833795 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:33:28.833763 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:33:30.044409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:30.044375 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.053972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:30.053950 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg"] Oct 30 01:33:30.065426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:30.065402 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-qmlqg"] Oct 30 01:33:30.225024 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:30.224980 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:31.874768 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:31.874730 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3460fc9-e075-4df7-983a-7a3c30ec199e" path="/var/lib/kubelet/pods/b3460fc9-e075-4df7-983a-7a3c30ec199e/volumes" Oct 30 01:33:31.961425 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:31.961386 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:32.656446 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:32.656419 5938 scope.go:122] "RemoveContainer" containerID="6d24d597c06442e88bbbbdf60ab62e764d7f2a6cf70d868232aa597257536f13" Oct 30 01:33:32.665117 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:32.665096 5938 scope.go:122] "RemoveContainer" containerID="43607cf6aac232748757d3d59be2881a617e78fbc040b79f6b1713cc365b5aa6" Oct 30 01:33:32.672623 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:32.672601 5938 scope.go:122] "RemoveContainer" containerID="a9f6fe64d8815c3982d6ba85ca023c3fd33057d3ee4b0fdc55b498967bce5c84" Oct 30 01:33:33.787955 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:33.787906 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:35.452561 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:35.452525 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:37.244262 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:37.244215 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:38.989671 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:38.989590 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:40.883432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:40.883401 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:42.514497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:42.514459 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:42.950664 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:42.950630 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:43.147756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:43.147706 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:43.729772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:43.729738 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:43.907335 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:43.907295 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:44.103856 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:44.103757 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:44.316778 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:44.316745 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:45.367418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:45.367372 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:46.110336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:46.110276 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:47.810494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.810380 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:47.871357 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.871319 5938 generic.go:356] "Generic (PLEG): container finished" podID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerID="906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428" exitCode=0 Oct 30 01:33:47.876132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.876101 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerDied","Data":"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428"} Oct 30 01:33:47.876601 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.876581 5938 scope.go:122] "RemoveContainer" containerID="906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428" Oct 30 01:33:47.877926 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.877897 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:47.877998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.877966 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:47.878050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:47.877997 5938 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 01:33:48.878711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:48.878677 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerStarted","Data":"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1"} Oct 30 01:33:49.069060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:49.069015 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:49.069271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:49.069249 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:49.069332 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:49.069296 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:49.459740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:49.459699 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:50.068809 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:50.068773 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:50.069258 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:50.068827 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:51.069267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:51.069232 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:51.069674 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:51.069282 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:51.181494 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:51.181462 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:52.069072 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:52.069037 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:52.069268 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:52.069108 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:52.799887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:52.798368 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:53.069286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:53.069205 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:53.069286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:53.069264 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:54.069050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:54.069010 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Oct 30 01:33:54.069517 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:54.069070 5938 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Oct 30 01:33:54.498178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:54.498150 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:55.073343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:55.073307 5938 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:55.073744 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:55.073726 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:55.074346 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:55.074326 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:33:56.221173 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:56.221136 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:57.912946 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:57.912909 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:58.834180 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:33:58.834144 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:33:59.762896 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:33:59.762843 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:01.447663 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:01.447591 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.092590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:03.092557 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:04.695607 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.695563 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88"] Oct 30 01:34:04.699569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.699540 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.715271 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.715246 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:04.715508 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.715477 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:04.726875 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.726831 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:04.726993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.726925 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:34:04.728829 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.728801 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88"] Oct 30 01:34:04.765982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.765954 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.766091 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.765993 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctcp\" (UniqueName: \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.766139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.766093 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.866753 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.866726 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.866894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.866762 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctcp\" (UniqueName: \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.866894 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.866795 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.867364 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.867345 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.869493 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.869468 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.889617 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.889593 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctcp\" (UniqueName: \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\") pod \"maas-api-revoke-sub-c1b4153b-9ts88\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:04.905100 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:04.905068 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:05.013478 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:05.013410 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:05.312036 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:05.311965 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88"] Oct 30 01:34:05.315574 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:34:05.315549 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62dedb8_7dc2_4e9d_ba83_09214ca25fa2.slice/crio-339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27 WatchSource:0}: Error finding container 339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27: Status 404 returned error can't find the container with id 339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27 Oct 30 01:34:05.987127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:05.987090 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" event={"ID":"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2","Type":"ContainerStarted","Data":"fd42a6ad2e6197b8b451039b40f256f2e751f91b932c59ca6f5f9827f007170f"} Oct 30 01:34:05.987532 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:05.987174 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" event={"ID":"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2","Type":"ContainerStarted","Data":"339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27"} Oct 30 01:34:06.026399 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:06.026352 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" podStartSLOduration=2.026339303 podStartE2EDuration="2.026339303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:06.025145603 +0000 UTC m=+3698.265380716" watchObservedRunningTime="2026-10-30 01:34:06.026339303 +0000 UTC m=+3698.266574415" Oct 30 01:34:06.598333 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:06.598287 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:06.993713 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:06.993671 5938 generic.go:356] "Generic (PLEG): container finished" podID="f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" containerID="fd42a6ad2e6197b8b451039b40f256f2e751f91b932c59ca6f5f9827f007170f" exitCode=0 Oct 30 01:34:06.994187 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:06.993767 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" event={"ID":"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2","Type":"ContainerDied","Data":"fd42a6ad2e6197b8b451039b40f256f2e751f91b932c59ca6f5f9827f007170f"} Oct 30 01:34:08.133338 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.133311 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:08.200844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.200820 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\" (UniqueName: \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\") pod \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " Oct 30 01:34:08.200999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.200889 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\") pod \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " Oct 30 01:34:08.200999 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.200945 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\") pod \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\" (UID: \"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2\") " Oct 30 01:34:08.201317 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.201294 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca" pod "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" (UID: "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:08.202842 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.202818 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp" pod "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" (UID: "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2"). InnerVolumeSpecName "kube-api-access-tctcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:08.203010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.202818 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca" pod "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" (UID: "f62dedb8-7dc2-4e9d-ba83-09214ca25fa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:08.271716 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.271638 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:08.302800 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.302770 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctcp\" (UniqueName: \"kubernetes.io/projected/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-kube-api-access-tctcp\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:08.302944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.302803 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:08.302944 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:08.302821 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:09.009136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:09.009067 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" Oct 30 01:34:09.009136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:09.009094 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88" event={"ID":"f62dedb8-7dc2-4e9d-ba83-09214ca25fa2","Type":"ContainerDied","Data":"339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27"} Oct 30 01:34:09.009136 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:09.009137 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339f67ee734cc3ec641f85d58156656db6232660997580b8b681c37957f4ca27" Oct 30 01:34:09.947168 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:09.947128 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:11.608587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:11.608526 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:13.480688 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:13.480654 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:15.184227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.184188 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:15.526685 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.526595 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:15.654122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.654091 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz"] Oct 30 01:34:15.654503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.654489 5938 cpu_manager.go:414] "containerMap: removing container" podUID="f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" containerName="revoke-keys" Oct 30 01:34:15.654503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.654503 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" containerName="revoke-keys" Oct 30 01:34:15.654610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.654570 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" containerName="revoke-keys" Oct 30 01:34:15.657785 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.657766 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.673610 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.673584 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:15.685132 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.685108 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:15.685267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.685216 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:15.685267 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.685230 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:34:15.686903 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.686855 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz"] Oct 30 01:34:15.768356 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.768327 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.768474 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.768385 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhv2\" (UniqueName: \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.768548 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.768516 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.869468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.869447 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.869576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.869485 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhv2\" (UniqueName: \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.869576 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.869528 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.870115 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.870094 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.872094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.872074 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.891689 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.891659 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhv2\" (UniqueName: \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:15.970609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:15.970587 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:16.289316 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:16.288949 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz"] Oct 30 01:34:16.983884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:16.983827 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:17.051389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:17.051355 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" event={"ID":"184e09f8-e37e-49ed-ae20-01d17f405734","Type":"ContainerStarted","Data":"6ed2c7a1b0b1a98bed236076ffebdf930edfbe4693dd1b4ccbc9bb7df5cd8904"} Oct 30 01:34:17.051585 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:17.051412 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" event={"ID":"184e09f8-e37e-49ed-ae20-01d17f405734","Type":"ContainerStarted","Data":"b5f073e249fe1e3924026e5655bb3b6a5cf80aab4946444049c41a89b0bd85bd"} Oct 30 01:34:17.094651 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:17.094588 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" podStartSLOduration=2.09456983 podStartE2EDuration="2.09456983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:17.091063844 +0000 UTC m=+3709.331298958" watchObservedRunningTime="2026-10-30 01:34:17.09456983 +0000 UTC m=+3709.334804944" Oct 30 01:34:18.058007 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:18.057971 5938 generic.go:356] "Generic (PLEG): container finished" podID="184e09f8-e37e-49ed-ae20-01d17f405734" containerID="6ed2c7a1b0b1a98bed236076ffebdf930edfbe4693dd1b4ccbc9bb7df5cd8904" exitCode=0 Oct 30 01:34:18.058395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:18.058059 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" event={"ID":"184e09f8-e37e-49ed-ae20-01d17f405734","Type":"ContainerDied","Data":"6ed2c7a1b0b1a98bed236076ffebdf930edfbe4693dd1b4ccbc9bb7df5cd8904"} Oct 30 01:34:18.675387 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:18.675348 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:19.194954 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.194925 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:19.308849 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.308825 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\" (UniqueName: \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\") pod \"184e09f8-e37e-49ed-ae20-01d17f405734\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " Oct 30 01:34:19.309010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.308939 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\") pod \"184e09f8-e37e-49ed-ae20-01d17f405734\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " Oct 30 01:34:19.309010 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.308973 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\") pod \"184e09f8-e37e-49ed-ae20-01d17f405734\" (UID: \"184e09f8-e37e-49ed-ae20-01d17f405734\") " Oct 30 01:34:19.309293 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.309272 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca" pod "184e09f8-e37e-49ed-ae20-01d17f405734" (UID: "184e09f8-e37e-49ed-ae20-01d17f405734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:19.310933 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.310911 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2" pod "184e09f8-e37e-49ed-ae20-01d17f405734" (UID: "184e09f8-e37e-49ed-ae20-01d17f405734"). InnerVolumeSpecName "kube-api-access-lzhv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:19.310993 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.310946 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca" pod "184e09f8-e37e-49ed-ae20-01d17f405734" (UID: "184e09f8-e37e-49ed-ae20-01d17f405734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:19.409904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.409856 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:19.409904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.409903 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184e09f8-e37e-49ed-ae20-01d17f405734-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:19.410046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:19.409915 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzhv2\" (UniqueName: \"kubernetes.io/projected/184e09f8-e37e-49ed-ae20-01d17f405734-kube-api-access-lzhv2\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:20.070094 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:20.070062 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" Oct 30 01:34:20.070259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:20.070093 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz" event={"ID":"184e09f8-e37e-49ed-ae20-01d17f405734","Type":"ContainerDied","Data":"b5f073e249fe1e3924026e5655bb3b6a5cf80aab4946444049c41a89b0bd85bd"} Oct 30 01:34:20.070259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:20.070144 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f073e249fe1e3924026e5655bb3b6a5cf80aab4946444049c41a89b0bd85bd" Oct 30 01:34:20.402740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:20.402697 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:20.611440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:20.611406 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:22.104497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:22.104463 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:23.867023 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:23.866994 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:25.508941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:25.508903 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:26.227453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:26.227420 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:26.375324 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:26.375289 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:27.214114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:27.214050 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:28.727503 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:28.727466 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:28.833551 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:34:28.833512 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:34:29.226718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.226680 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:29.315096 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.315064 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz"] Oct 30 01:34:29.328614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.328580 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fe8356-2xnzz"] Oct 30 01:34:29.369457 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.369422 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:29.852587 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.852552 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:29.875953 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:29.875916 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184e09f8-e37e-49ed-ae20-01d17f405734" path="/var/lib/kubelet/pods/184e09f8-e37e-49ed-ae20-01d17f405734/volumes" Oct 30 01:34:32.700359 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:32.700325 5938 scope.go:122] "RemoveContainer" containerID="d8749420c71670b67f4fd08faea3cf4976f8c865bf3257b2ba38d56782368c47" Oct 30 01:34:32.708783 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:32.708763 5938 scope.go:122] "RemoveContainer" containerID="dd9fd5247a22ab5d69e2955d0d3f1035cff105cbbe7c5c747896d91a723e916a" Oct 30 01:34:32.716650 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:32.716629 5938 scope.go:122] "RemoveContainer" containerID="ab928338d2cefc73a445205e69500ae4aaf1650efb35cd3499ba7d46cf95c52a" Oct 30 01:34:35.803711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:35.803672 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:35.951984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:35.951945 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:36.115756 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:36.115712 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:34:36.116292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:36.116259 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" containerID="cri-o://5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1" gracePeriod=30 Oct 30 01:34:36.200397 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:36.200360 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:40.077076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:40.077034 5938 patch_prober.go:29] interesting pod/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:34:40.077552 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:40.077104 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:34:41.376037 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.376005 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:34:41.395619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395545 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395576 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395619 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395602 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395655 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395695 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395764 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395791 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.395915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395823 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.396226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395890 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:41.396226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.395964 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:41.396226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396118 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:41.396226 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396155 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\") pod \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\" (UID: \"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2\") " Oct 30 01:34:41.396462 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396285 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:41.396599 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396578 5938 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.396717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396604 5938 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-data\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.396717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396618 5938 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-credential-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.396717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396634 5938 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-workload-socket\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.396717 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.396685 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:41.398899 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.398834 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:41.399013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.398957 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:34:41.399013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.398986 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "kube-api-access-m4wdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:41.399107 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.399087 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy" pod "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" (UID: "b4567cd8-e736-4cfe-96ca-ecac7d69b8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:41.458327 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.458303 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:41.497885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.497844 5938 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istiod-ca-cert\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.497984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.497893 5938 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-envoy\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.497984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.497904 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wdc\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-kube-api-access-m4wdc\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.497984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.497916 5938 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-token\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:41.497984 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:41.497926 5938 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2-istio-podinfo\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:34:42.185694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.185624 5938 generic.go:356] "Generic (PLEG): container finished" podID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerID="5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1" exitCode=0 Oct 30 01:34:42.185694 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.185674 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" Oct 30 01:34:42.185914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.185707 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerDied","Data":"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1"} Oct 30 01:34:42.185914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.185764 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk" event={"ID":"b4567cd8-e736-4cfe-96ca-ecac7d69b8e2","Type":"ContainerDied","Data":"8779dc04e93cb4459c9a19d69354830a00b9e8f832df68b0c6ffc87f9d7c706a"} Oct 30 01:34:42.185914 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.185787 5938 scope.go:122] "RemoveContainer" containerID="5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1" Oct 30 01:34:42.194879 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.194842 5938 scope.go:122] "RemoveContainer" containerID="906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428" Oct 30 01:34:42.202750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.202733 5938 scope.go:122] "RemoveContainer" containerID="5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1" Oct 30 01:34:42.203043 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:34:42.203025 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1\": container with ID starting with 5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1 not found: ID does not exist" containerID="5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1" Oct 30 01:34:42.203114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.203051 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1"} err="failed to get container status \"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1\": rpc error: code = NotFound desc = could not find container \"5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1\": container with ID starting with 5c26fc08131b60e60ea49eac24c0e7db1f480efc8c40108af891bf2ebdb004d1 not found: ID does not exist" Oct 30 01:34:42.203114 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.203066 5938 scope.go:122] "RemoveContainer" containerID="906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428" Oct 30 01:34:42.203331 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:34:42.203312 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428\": container with ID starting with 906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428 not found: ID does not exist" containerID="906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428" Oct 30 01:34:42.203382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.203337 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428"} err="failed to get container status \"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428\": rpc error: code = NotFound desc = could not find container \"906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428\": container with ID starting with 906f997b6d6019d98fd0e5a26bdc0e8973f127c0375db377c4adb2704fc62428 not found: ID does not exist" Oct 30 01:34:42.247196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.247173 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:34:42.257731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:42.257710 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-fe8356-openshift-default-586db97f7-zm8jk"] Oct 30 01:34:43.092292 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:43.092254 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:43.233178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:43.233145 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:43.875540 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:43.875508 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" path="/var/lib/kubelet/pods/b4567cd8-e736-4cfe-96ca-ecac7d69b8e2/volumes" Oct 30 01:34:56.321180 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:56.321143 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:57.044418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:57.044381 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.026360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:34:58.026326 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.834662 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:34:58.834624 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:35:02.675887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:02.674982 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-lvppb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:35:02.675887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:02.675076 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-lvppb" podUID="e2491bdb-6953-40e8-8c95-5c39bca71b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:35:02.716054 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:02.716008 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-ccb2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:35:02.716348 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:02.716303 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-ccb2j" podUID="04691eac-59b2-40f2-bea3-84cba2e43e17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:35:12.401398 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:12.401359 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:20.856460 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.856414 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv"] Oct 30 01:35:20.857062 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857038 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:20.857156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857064 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:20.857156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857095 5938 cpu_manager.go:414] "containerMap: removing container" podUID="184e09f8-e37e-49ed-ae20-01d17f405734" containerName="revoke-keys" Oct 30 01:35:20.857156 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857104 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e09f8-e37e-49ed-ae20-01d17f405734" containerName="revoke-keys" Oct 30 01:35:20.857375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857225 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:20.857375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857243 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:20.857375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.857256 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e09f8-e37e-49ed-ae20-01d17f405734" containerName="revoke-keys" Oct 30 01:35:20.860904 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.860879 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:20.876888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.875509 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:20.896728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.894141 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:20.896728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.894604 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:20.896728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.894691 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:35:20.896728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.896643 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv"] Oct 30 01:35:20.955636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.955605 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:20.955791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.955652 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:20.955791 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:20.955682 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhv56\" (UniqueName: \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.057331 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.057295 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.057497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.057341 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.057497 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.057369 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhv56\" (UniqueName: \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.058515 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.058485 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.060777 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.060747 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.080729 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.080701 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhv56\" (UniqueName: \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\") pod \"maas-api-revoke-sub-452b6d7b-65qcv\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.180354 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.180281 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:21.507509 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:21.507423 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv"] Oct 30 01:35:21.511934 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:35:21.511902 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5053c4_3109_44ec_ab03_6ef5e13dd9a5.slice/crio-8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075 WatchSource:0}: Error finding container 8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075: Status 404 returned error can't find the container with id 8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075 Oct 30 01:35:22.484189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:22.484073 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" event={"ID":"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5","Type":"ContainerStarted","Data":"45a425d94334a56b51c50441756b37026e2533ffde2d74d73b17856ab9d66218"} Oct 30 01:35:22.484189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:22.484143 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" event={"ID":"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5","Type":"ContainerStarted","Data":"8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075"} Oct 30 01:35:22.540231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:22.540169 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" podStartSLOduration=2.54015096 podStartE2EDuration="2.54015096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:22.539111809 +0000 UTC m=+3774.779346923" watchObservedRunningTime="2026-10-30 01:35:22.54015096 +0000 UTC m=+3774.780386074" Oct 30 01:35:23.527888 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:23.524584 5938 generic.go:356] "Generic (PLEG): container finished" podID="8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" containerID="45a425d94334a56b51c50441756b37026e2533ffde2d74d73b17856ab9d66218" exitCode=0 Oct 30 01:35:23.528590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:23.528517 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" event={"ID":"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5","Type":"ContainerDied","Data":"45a425d94334a56b51c50441756b37026e2533ffde2d74d73b17856ab9d66218"} Oct 30 01:35:24.685291 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.685266 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:24.748765 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.748742 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\" (UniqueName: \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\") pod \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " Oct 30 01:35:24.748920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.748774 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\") pod \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " Oct 30 01:35:24.748920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.748851 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\") pod \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\" (UID: \"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5\") " Oct 30 01:35:24.749334 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.749302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca" pod "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" (UID: "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:24.750827 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.750799 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56" pod "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" (UID: "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5"). InnerVolumeSpecName "kube-api-access-qhv56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:24.750931 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.750880 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca" pod "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" (UID: "8f5053c4-3109-44ec-ab03-6ef5e13dd9a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:24.849748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.849684 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhv56\" (UniqueName: \"kubernetes.io/projected/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-kube-api-access-qhv56\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:24.849748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.849717 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:24.849748 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:24.849727 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5053c4-3109-44ec-ab03-6ef5e13dd9a5-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:25.553569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:25.553535 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" Oct 30 01:35:25.553569 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:25.553552 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-65qcv" event={"ID":"8f5053c4-3109-44ec-ab03-6ef5e13dd9a5","Type":"ContainerDied","Data":"8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075"} Oct 30 01:35:25.553772 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:25.553600 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce810e0e5f0d13d4f5b4fcea266f8636b24c1847c88c06a3bc24e8dfc5a8075" Oct 30 01:35:28.834455 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:35:28.834396 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:35:31.036725 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:31.036687 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z"] Oct 30 01:35:31.048061 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:31.048027 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a417f94-cfm8z"] Oct 30 01:35:31.876733 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:31.876693 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e991ab6e-9ef0-423c-b530-d727fad65144" path="/var/lib/kubelet/pods/e991ab6e-9ef0-423c-b530-d727fad65144/volumes" Oct 30 01:35:32.239476 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:32.239397 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:32.747404 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:32.747374 5938 scope.go:122] "RemoveContainer" containerID="557313dc6f82234077cc46a85b338a73beb06410cf45ed46a011a54085917126" Oct 30 01:35:32.755450 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:32.755434 5938 scope.go:122] "RemoveContainer" containerID="9eec52aa966fb5a3c41e31853187ef802fff574f97d3fafe2d0944f9a9a2cbb9" Oct 30 01:35:32.762912 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:32.762888 5938 scope.go:122] "RemoveContainer" containerID="77f07f3a3b53fa39b232fe88f6ab212cc2e6ab157b8e2a23129cb58b122a32e2" Oct 30 01:35:33.533109 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:33.533078 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:33.681626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:33.681596 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:37.837076 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:37.837036 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:41.470902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.468479 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb"] Oct 30 01:35:41.472043 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.472015 5938 cpu_manager.go:414] "containerMap: removing container" podUID="8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" containerName="revoke-keys" Oct 30 01:35:41.472227 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.472209 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" containerName="revoke-keys" Oct 30 01:35:41.472385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.472370 5938 cpu_manager.go:414] "containerMap: removing container" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:41.472495 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.472481 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4567cd8-e736-4cfe-96ca-ecac7d69b8e2" containerName="istio-proxy" Oct 30 01:35:41.472740 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.472722 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5053c4-3109-44ec-ab03-6ef5e13dd9a5" containerName="revoke-keys" Oct 30 01:35:41.487179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.487159 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.491880 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.491838 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb"] Oct 30 01:35:41.515302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.515280 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:41.515534 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.515516 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:41.531066 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.531044 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:35:41.531280 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.531265 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:41.532247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.532213 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.532516 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.532497 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.532711 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.532693 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdk4j\" (UniqueName: \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.641320 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.641288 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.658670 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.642244 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.677965 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.677937 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.678161 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.678142 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdk4j\" (UniqueName: \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.684884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.683522 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.742382 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.742322 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdk4j\" (UniqueName: \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\") pod \"maas-api-revoke-sub-3e31a506-vxhkb\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:41.854885 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:41.853975 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:42.720887 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:42.699801 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb"] Oct 30 01:35:42.833025 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:35:42.832994 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb294a23_7581_429e_956b_3d170a28bc46.slice/crio-b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9 WatchSource:0}: Error finding container b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9: Status 404 returned error can't find the container with id b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9 Oct 30 01:35:43.238886 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:43.236126 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:43.761336 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:43.761294 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" event={"ID":"cb294a23-7581-429e-956b-3d170a28bc46","Type":"ContainerStarted","Data":"01d40603d87918f8ce378ca9b59a5d6239300101a07573ba9567ced2ed30b4f8"} Oct 30 01:35:43.761750 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:43.761395 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" event={"ID":"cb294a23-7581-429e-956b-3d170a28bc46","Type":"ContainerStarted","Data":"b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9"} Oct 30 01:35:43.800485 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:43.800436 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" podStartSLOduration=2.800423581 podStartE2EDuration="2.800423581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:43.799166168 +0000 UTC m=+3796.039401282" watchObservedRunningTime="2026-10-30 01:35:43.800423581 +0000 UTC m=+3796.040658693" Oct 30 01:35:44.768727 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:44.768686 5938 generic.go:356] "Generic (PLEG): container finished" podID="cb294a23-7581-429e-956b-3d170a28bc46" containerID="01d40603d87918f8ce378ca9b59a5d6239300101a07573ba9567ced2ed30b4f8" exitCode=0 Oct 30 01:35:44.769158 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:44.768738 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" event={"ID":"cb294a23-7581-429e-956b-3d170a28bc46","Type":"ContainerDied","Data":"01d40603d87918f8ce378ca9b59a5d6239300101a07573ba9567ced2ed30b4f8"} Oct 30 01:35:45.922614 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:45.922588 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:46.000821 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.000792 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\") pod \"cb294a23-7581-429e-956b-3d170a28bc46\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " Oct 30 01:35:46.000975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.000929 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\" (UniqueName: \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\") pod \"cb294a23-7581-429e-956b-3d170a28bc46\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " Oct 30 01:35:46.000975 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.000970 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\") pod \"cb294a23-7581-429e-956b-3d170a28bc46\" (UID: \"cb294a23-7581-429e-956b-3d170a28bc46\") " Oct 30 01:35:46.001254 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.001229 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca" pod "cb294a23-7581-429e-956b-3d170a28bc46" (UID: "cb294a23-7581-429e-956b-3d170a28bc46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:46.002928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.002907 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j" pod "cb294a23-7581-429e-956b-3d170a28bc46" (UID: "cb294a23-7581-429e-956b-3d170a28bc46"). InnerVolumeSpecName "kube-api-access-kdk4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:46.003004 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.002951 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca" pod "cb294a23-7581-429e-956b-3d170a28bc46" (UID: "cb294a23-7581-429e-956b-3d170a28bc46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:46.101916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.101895 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:46.102005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.101917 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdk4j\" (UniqueName: \"kubernetes.io/projected/cb294a23-7581-429e-956b-3d170a28bc46-kube-api-access-kdk4j\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:46.102005 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.101927 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb294a23-7581-429e-956b-3d170a28bc46-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:35:46.780060 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.780030 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" Oct 30 01:35:46.780207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.780057 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vxhkb" event={"ID":"cb294a23-7581-429e-956b-3d170a28bc46","Type":"ContainerDied","Data":"b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9"} Oct 30 01:35:46.780207 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:35:46.780109 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b610d620c42cd08bb44cf496247f5b3090975ce81600de8b4469d45e520427e9" Oct 30 01:35:58.979359 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:35:58.979303 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:36:03.119188 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:03.119151 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:05.028554 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.028515 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk"] Oct 30 01:36:05.039453 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.039425 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r5wrk"] Oct 30 01:36:05.063074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.063049 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw"] Oct 30 01:36:05.063473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.063453 5938 cpu_manager.go:414] "containerMap: removing container" podUID="cb294a23-7581-429e-956b-3d170a28bc46" containerName="revoke-keys" Oct 30 01:36:05.063473 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.063469 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb294a23-7581-429e-956b-3d170a28bc46" containerName="revoke-keys" Oct 30 01:36:05.063626 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.063539 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb294a23-7581-429e-956b-3d170a28bc46" containerName="revoke-keys" Oct 30 01:36:05.066360 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.066343 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.083995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.083971 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:36:05.083995 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.083980 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:36:05.084201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.084005 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:36:05.084201 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.084025 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5phsh\"" Oct 30 01:36:05.085286 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.085258 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw"] Oct 30 01:36:05.181491 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.181460 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.181636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.181495 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6x6g\" (UniqueName: \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.181684 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.181644 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.282436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.282377 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.282436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.282432 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.282630 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.282453 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6x6g\" (UniqueName: \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.283198 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.283176 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.284707 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.284685 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.304808 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.304781 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6x6g\" (UniqueName: \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\") pod \"maas-api-revoke-sub-47dd818d-sdcpw\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.380968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.380946 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:05.676475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.676336 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw"] Oct 30 01:36:05.679306 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:36:05.679276 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f0f492_eeac_4116_8daf_1e4fd5e76104.slice/crio-d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12 WatchSource:0}: Error finding container d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12: Status 404 returned error can't find the container with id d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12 Oct 30 01:36:05.874353 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.874326 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37907747-6761-44a7-9710-509e69fe0e04" path="/var/lib/kubelet/pods/37907747-6761-44a7-9710-509e69fe0e04/volumes" Oct 30 01:36:05.915396 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:05.915373 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" event={"ID":"d8f0f492-eeac-4116-8daf-1e4fd5e76104","Type":"ContainerStarted","Data":"d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12"} Oct 30 01:36:06.921712 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:06.921682 5938 generic.go:356] "Generic (PLEG): container finished" podID="d8f0f492-eeac-4116-8daf-1e4fd5e76104" containerID="4efa79c971e92bd89231f2f3130554945aac044779ca64923348c8103c86afed" exitCode=0 Oct 30 01:36:06.922121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:06.921730 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" event={"ID":"d8f0f492-eeac-4116-8daf-1e4fd5e76104","Type":"ContainerDied","Data":"4efa79c971e92bd89231f2f3130554945aac044779ca64923348c8103c86afed"} Oct 30 01:36:08.067522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.067496 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:08.218102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.218047 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\") pod \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " Oct 30 01:36:08.218102 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.218087 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\" (UniqueName: \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\") pod \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " Oct 30 01:36:08.218275 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.218211 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\") pod \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\" (UID: \"d8f0f492-eeac-4116-8daf-1e4fd5e76104\") " Oct 30 01:36:08.218434 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.218411 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca" pod "d8f0f492-eeac-4116-8daf-1e4fd5e76104" (UID: "d8f0f492-eeac-4116-8daf-1e4fd5e76104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:08.218537 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.218518 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-openshift-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:08.220045 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.220023 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g" pod "d8f0f492-eeac-4116-8daf-1e4fd5e76104" (UID: "d8f0f492-eeac-4116-8daf-1e4fd5e76104"). InnerVolumeSpecName "kube-api-access-w6x6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:08.220139 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.220064 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca" pod "d8f0f492-eeac-4116-8daf-1e4fd5e76104" (UID: "d8f0f492-eeac-4116-8daf-1e4fd5e76104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:08.319163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.319140 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f0f492-eeac-4116-8daf-1e4fd5e76104-maas-api-xks-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:08.319163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.319161 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6x6g\" (UniqueName: \"kubernetes.io/projected/d8f0f492-eeac-4116-8daf-1e4fd5e76104-kube-api-access-w6x6g\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:08.934179 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.934151 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" Oct 30 01:36:08.934343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.934175 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47dd818d-sdcpw" event={"ID":"d8f0f492-eeac-4116-8daf-1e4fd5e76104","Type":"ContainerDied","Data":"d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12"} Oct 30 01:36:08.934343 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:08.934229 5938 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22d9cabb00a95627cae18a3ea89784b986400c7a7c33413ea87e362bad05f12" Oct 30 01:36:17.417893 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.417849 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:17.418918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.418887 5938 cpu_manager.go:414] "containerMap: removing container" podUID="d8f0f492-eeac-4116-8daf-1e4fd5e76104" containerName="revoke-keys" Oct 30 01:36:17.418918 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.418913 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f0f492-eeac-4116-8daf-1e4fd5e76104" containerName="revoke-keys" Oct 30 01:36:17.419127 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.418991 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f0f492-eeac-4116-8daf-1e4fd5e76104" containerName="revoke-keys" Oct 30 01:36:17.422074 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.422051 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.441884 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.441844 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:17.454281 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.454245 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:36:17.454483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.454270 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x95xl\"" Oct 30 01:36:17.454483 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.454297 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 01:36:17.454681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.454313 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:36:17.454681 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.454367 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:36:17.597440 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.597406 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqcn\" (UniqueName: \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.597631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.597448 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.597631 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.597480 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.597771 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.597659 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.698477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.698404 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqcn\" (UniqueName: \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.698477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.698441 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.698477 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.698472 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.698703 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.698504 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.699282 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.699255 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.701044 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.701010 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.701181 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.701128 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.720921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.720896 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqcn\" (UniqueName: \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\") pod \"maas-api-5c48877c75-cr2s9\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:17.737190 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:17.737166 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:18.035521 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:18.035372 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:18.038289 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:36:18.038262 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e75580_3fbc_42fc_a2f2_35af31faa68e.slice/crio-f96b3bab15108e4b986bce1caa3bed541708b33a78b682c38d689ff23cd8e694 WatchSource:0}: Error finding container f96b3bab15108e4b986bce1caa3bed541708b33a78b682c38d689ff23cd8e694: Status 404 returned error can't find the container with id f96b3bab15108e4b986bce1caa3bed541708b33a78b682c38d689ff23cd8e694 Oct 30 01:36:18.039420 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:18.039395 5938 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:36:18.981668 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:18.981631 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" event={"ID":"97e75580-3fbc-42fc-a2f2-35af31faa68e","Type":"ContainerStarted","Data":"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60"} Oct 30 01:36:18.982110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:18.981688 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" event={"ID":"97e75580-3fbc-42fc-a2f2-35af31faa68e","Type":"ContainerStarted","Data":"f96b3bab15108e4b986bce1caa3bed541708b33a78b682c38d689ff23cd8e694"} Oct 30 01:36:18.982110 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:18.981726 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:19.025432 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:19.025387 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" podStartSLOduration=1.647370965 podStartE2EDuration="2.02537437s" totalImagesPullingTime="378.003405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:19.024064676 +0000 UTC m=+3831.264299790" watchObservedRunningTime="2026-10-30 01:36:19.02537437 +0000 UTC m=+3831.265609477" Oct 30 01:36:24.697544 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.697507 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:24.698055 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.697679 5938 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" podUID="97e75580-3fbc-42fc-a2f2-35af31faa68e" containerName="maas-api" containerID="cri-o://c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60" gracePeriod=30 Oct 30 01:36:24.702915 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.702894 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:24.730693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.730668 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-d554x"] Oct 30 01:36:24.734206 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.734188 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.752891 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.752845 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxztv\" (UniqueName: \"kubernetes.io/projected/7876ad2b-a912-4ba4-9542-e7404db96193-kube-api-access-sxztv\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.753082 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.753057 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-maas-api-tls\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.753384 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.753353 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7876ad2b-a912-4ba4-9542-e7404db96193-openshift-service-ca\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.753475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.753390 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-metrics-certs\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.754761 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.754741 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-d554x"] Oct 30 01:36:24.854050 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.853996 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxztv\" (UniqueName: \"kubernetes.io/projected/7876ad2b-a912-4ba4-9542-e7404db96193-kube-api-access-sxztv\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.854216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.854146 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-maas-api-tls\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.854302 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.854286 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7876ad2b-a912-4ba4-9542-e7404db96193-openshift-service-ca\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.854345 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.854326 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-metrics-certs\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.854982 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.854934 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7876ad2b-a912-4ba4-9542-e7404db96193-openshift-service-ca\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.856604 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.856573 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-maas-api-tls\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.856728 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.856682 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7876ad2b-a912-4ba4-9542-e7404db96193-metrics-certs\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.877844 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.877818 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxztv\" (UniqueName: \"kubernetes.io/projected/7876ad2b-a912-4ba4-9542-e7404db96193-kube-api-access-sxztv\") pod \"maas-api-595cc648c-d554x\" (UID: \"7876ad2b-a912-4ba4-9542-e7404db96193\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:24.942654 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.942632 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:24.955218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.955178 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\" (UniqueName: \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\") pod \"97e75580-3fbc-42fc-a2f2-35af31faa68e\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " Oct 30 01:36:24.955218 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.955208 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\") pod \"97e75580-3fbc-42fc-a2f2-35af31faa68e\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " Oct 30 01:36:24.955385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.955239 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\") pod \"97e75580-3fbc-42fc-a2f2-35af31faa68e\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " Oct 30 01:36:24.955385 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.955291 5938 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\") pod \"97e75580-3fbc-42fc-a2f2-35af31faa68e\" (UID: \"97e75580-3fbc-42fc-a2f2-35af31faa68e\") " Oct 30 01:36:24.955669 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.955641 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca" pod "97e75580-3fbc-42fc-a2f2-35af31faa68e" (UID: "97e75580-3fbc-42fc-a2f2-35af31faa68e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:24.957133 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.957104 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn" pod "97e75580-3fbc-42fc-a2f2-35af31faa68e" (UID: "97e75580-3fbc-42fc-a2f2-35af31faa68e"). InnerVolumeSpecName "kube-api-access-qhqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:24.957276 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.957248 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs" pod "97e75580-3fbc-42fc-a2f2-35af31faa68e" (UID: "97e75580-3fbc-42fc-a2f2-35af31faa68e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:24.957370 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:24.957302 5938 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls" pod "97e75580-3fbc-42fc-a2f2-35af31faa68e" (UID: "97e75580-3fbc-42fc-a2f2-35af31faa68e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:25.012636 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.012611 5938 generic.go:356] "Generic (PLEG): container finished" podID="97e75580-3fbc-42fc-a2f2-35af31faa68e" containerID="c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60" exitCode=0 Oct 30 01:36:25.012763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.012678 5938 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" Oct 30 01:36:25.012763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.012687 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" event={"ID":"97e75580-3fbc-42fc-a2f2-35af31faa68e","Type":"ContainerDied","Data":"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60"} Oct 30 01:36:25.012763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.012733 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9" event={"ID":"97e75580-3fbc-42fc-a2f2-35af31faa68e","Type":"ContainerDied","Data":"f96b3bab15108e4b986bce1caa3bed541708b33a78b682c38d689ff23cd8e694"} Oct 30 01:36:25.012763 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.012757 5938 scope.go:122] "RemoveContainer" containerID="c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60" Oct 30 01:36:25.021362 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.021343 5938 scope.go:122] "RemoveContainer" containerID="c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60" Oct 30 01:36:25.021685 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:36:25.021665 5938 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60\": container with ID starting with c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60 not found: ID does not exist" containerID="c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60" Oct 30 01:36:25.021774 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.021693 5938 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60"} err="failed to get container status \"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60\": rpc error: code = NotFound desc = could not find container \"c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60\": container with ID starting with c0988e40f86dfd7a8fa45d6bc45e1bab60decf1a214515bfe6f26a9f67a70d60 not found: ID does not exist" Oct 30 01:36:25.057014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.056987 5938 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqcn\" (UniqueName: \"kubernetes.io/projected/97e75580-3fbc-42fc-a2f2-35af31faa68e-kube-api-access-qhqcn\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:25.057014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.057012 5938 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-metrics-certs\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:25.057160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.057027 5938 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e75580-3fbc-42fc-a2f2-35af31faa68e-openshift-service-ca\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:25.057160 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.057039 5938 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e75580-3fbc-42fc-a2f2-35af31faa68e-maas-api-tls\") on node \"ip-10-0-131-130.ec2.internal\" DevicePath \"\"" Oct 30 01:36:25.060423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.060400 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:25.064648 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.064625 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-cr2s9"] Oct 30 01:36:25.107786 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.107767 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:25.402300 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.402275 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-d554x"] Oct 30 01:36:25.405011 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:36:25.404988 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7876ad2b_a912_4ba4_9542_e7404db96193.slice/crio-00747798d75c506d85a6a6b8890c358868903428b6c0e45c40ea8f853a9fa417 WatchSource:0}: Error finding container 00747798d75c506d85a6a6b8890c358868903428b6c0e45c40ea8f853a9fa417: Status 404 returned error can't find the container with id 00747798d75c506d85a6a6b8890c358868903428b6c0e45c40ea8f853a9fa417 Oct 30 01:36:25.720833 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.720803 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:25.874172 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.874144 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e75580-3fbc-42fc-a2f2-35af31faa68e" path="/var/lib/kubelet/pods/97e75580-3fbc-42fc-a2f2-35af31faa68e/volumes" Oct 30 01:36:25.921111 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:25.921083 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:26.018520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:26.018451 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" event={"ID":"7876ad2b-a912-4ba4-9542-e7404db96193","Type":"ContainerStarted","Data":"5a61efb0370845032dac5d2d7144b4867ef1edcb944e1e6271c4c05b5ce273fe"} Oct 30 01:36:26.018520 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:26.018510 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" event={"ID":"7876ad2b-a912-4ba4-9542-e7404db96193","Type":"ContainerStarted","Data":"00747798d75c506d85a6a6b8890c358868903428b6c0e45c40ea8f853a9fa417"} Oct 30 01:36:26.018693 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:26.018546 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:26.059921 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:26.059858 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" podStartSLOduration=1.696339185 podStartE2EDuration="2.059844775s" totalImagesPullingTime="363.50559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:26.058421913 +0000 UTC m=+3838.298657029" watchObservedRunningTime="2026-10-30 01:36:26.059844775 +0000 UTC m=+3838.300079887" Oct 30 01:36:28.998886 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:36:28.998562 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:36:32.032013 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:32.031984 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-d554x" Oct 30 01:36:32.792409 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:32.792377 5938 scope.go:122] "RemoveContainer" containerID="7bbbfc5ed31a27928d76548ea8ad37c1911751048b219b9430d11eb19590b1d2" Oct 30 01:36:43.366189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:36:43.366153 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:58.834635 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:36:58.834590 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:37:13.457196 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:13.457121 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797fb9856f-8rq9w_cacb0b7a-9b96-4964-b8cc-cab7a5c17e43/authorino/0.log" Oct 30 01:37:15.045034 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:15.044985 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kgxxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:15.045695 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:15.045644 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kgxxw" podUID="1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:15.309031 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:15.308942 5938 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-srsm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:15.309285 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:15.309254 5938 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-srsm4" podUID="b20309a7-fa26-440e-80bb-955b9fac548b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:25.077983 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:25.077948 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-xjxz2_3c55b307-9d39-4398-bf90-5dcaa5e22d15/manager/0.log" Oct 30 01:37:25.749426 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:25.749399 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-jqwwk_8afdb479-e42f-47c4-bd9c-684b2d2a3b85/copy-manifests/0.log" Oct 30 01:37:25.780652 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:25.780626 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-jqwwk_8afdb479-e42f-47c4-bd9c-684b2d2a3b85/manager/0.log" Oct 30 01:37:27.211602 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:27.211575 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-d554x_7876ad2b-a912-4ba4-9542-e7404db96193/maas-api/0.log" Oct 30 01:37:27.604444 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:27.604419 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vxhkb_cb294a23-7581-429e-956b-3d170a28bc46/revoke-keys/0.log" Oct 30 01:37:27.786454 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:27.786422 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-65qcv_8f5053c4-3109-44ec-ab03-6ef5e13dd9a5/revoke-keys/0.log" Oct 30 01:37:27.961628 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:27.961553 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47dd818d-sdcpw_d8f0f492-eeac-4116-8daf-1e4fd5e76104/revoke-keys/0.log" Oct 30 01:37:28.330202 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:28.330125 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c1b4153b-9ts88_f62dedb8-7dc2-4e9d-ba83-09214ca25fa2/revoke-keys/0.log" Oct 30 01:37:28.505972 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:28.505946 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ls2fc_a31b2986-29c6-49a2-95db-40e51c5ce410/revoke-keys/0.log" Oct 30 01:37:28.692924 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:28.692896 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-mq5dz_11b76a46-a496-460c-be8e-0147ba6cfe01/revoke-keys/0.log" Oct 30 01:37:28.833591 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:37:28.833557 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:37:30.557365 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:30.557332 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797fb9856f-8rq9w_cacb0b7a-9b96-4964-b8cc-cab7a5c17e43/authorino/0.log" Oct 30 01:37:30.764216 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:30.764194 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:37:30.804349 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:30.804331 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8mdqj_34e4db94-de44-4a89-9bfe-5ac1d5997038/ovn-acl-logging/0.log" Oct 30 01:37:31.824375 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:31.824342 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n6n52_8098738c-5d92-4b27-9196-c0071d6590e4/manager/0.log" Oct 30 01:37:32.817977 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:32.817951 5938 scope.go:122] "RemoveContainer" containerID="6a4f48a885d9a6dd4fc011d7c4544352a0f7b866578788ee9bada71992a77532" Oct 30 01:37:32.826427 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:32.826397 5938 scope.go:122] "RemoveContainer" containerID="37b4be80c0dccf31e36b2b330175cf6fe6fb20754b839326fd60d9e4d5a83c16" Oct 30 01:37:33.155122 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:33.155089 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-w85dq_b2695552-6541-4a4b-90ab-c26a57b3c570/kube-auth-proxy/0.log" Oct 30 01:37:33.934754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:33.934720 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-ccb2j_04691eac-59b2-40f2-bea3-84cba2e43e17/istio-proxy/0.log" Oct 30 01:37:34.518479 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:34.518431 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kgxxw_1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0/istio-proxy/0.log" Oct 30 01:37:34.717152 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:34.717112 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-lvppb_e2491bdb-6953-40e8-8c95-5c39bca71b56/istio-proxy/0.log" Oct 30 01:37:35.095163 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:35.095135 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-srsm4_b20309a7-fa26-440e-80bb-955b9fac548b/istio-proxy/0.log" Oct 30 01:37:35.295773 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:35.295743 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qcfkv_97cce19f-6ecc-4cf9-8205-628cf08ec662/payload-pre-processing/0.log" Oct 30 01:37:36.847710 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:36.847679 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jrvcb_f8099822-8f87-43de-9860-0d34402a1669/main/0.log" Oct 30 01:37:37.046380 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:37.046352 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wpnlg_8a59a208-000a-448d-9ca1-6042908e69bb/main/0.log" Oct 30 01:37:37.431150 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:37.431125 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv_111fbfd3-f42e-4d29-b107-65ddfa99c0a4/main/0.log" Oct 30 01:37:37.789468 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:37.789398 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d_5826cc58-3128-42cc-b1e6-06847d5ed3b2/main/0.log" Oct 30 01:37:38.651968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:38.651943 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797fb9856f-8rq9w_cacb0b7a-9b96-4964-b8cc-cab7a5c17e43/authorino/0.log" Oct 30 01:37:43.620378 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:43.620341 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:49.384718 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:49.384685 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-xjxz2_3c55b307-9d39-4398-bf90-5dcaa5e22d15/manager/0.log" Oct 30 01:37:50.143942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:50.143908 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-jqwwk_8afdb479-e42f-47c4-bd9c-684b2d2a3b85/copy-manifests/0.log" Oct 30 01:37:50.175439 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:50.175413 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-jqwwk_8afdb479-e42f-47c4-bd9c-684b2d2a3b85/manager/0.log" Oct 30 01:37:51.439724 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:51.439697 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-d554x_7876ad2b-a912-4ba4-9542-e7404db96193/maas-api/0.log" Oct 30 01:37:51.853395 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:51.853368 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-vxhkb_cb294a23-7581-429e-956b-3d170a28bc46/revoke-keys/0.log" Oct 30 01:37:52.036475 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:52.036451 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-65qcv_8f5053c4-3109-44ec-ab03-6ef5e13dd9a5/revoke-keys/0.log" Oct 30 01:37:52.236519 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:52.236432 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-47dd818d-sdcpw_d8f0f492-eeac-4116-8daf-1e4fd5e76104/revoke-keys/0.log" Oct 30 01:37:52.651118 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:52.651094 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c1b4153b-9ts88_f62dedb8-7dc2-4e9d-ba83-09214ca25fa2/revoke-keys/0.log" Oct 30 01:37:52.742957 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:52.742919 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:52.853680 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:52.853659 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-ls2fc_a31b2986-29c6-49a2-95db-40e51c5ce410/revoke-keys/0.log" Oct 30 01:37:53.042731 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:53.042666 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-mq5dz_11b76a46-a496-460c-be8e-0147ba6cfe01/revoke-keys/0.log" Oct 30 01:37:54.583259 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:54.583177 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797fb9856f-8rq9w_cacb0b7a-9b96-4964-b8cc-cab7a5c17e43/authorino/0.log" Oct 30 01:37:55.986157 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:55.986122 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n6n52_8098738c-5d92-4b27-9196-c0071d6590e4/manager/0.log" Oct 30 01:37:57.137759 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:57.137732 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-w85dq_b2695552-6541-4a4b-90ab-c26a57b3c570/kube-auth-proxy/0.log" Oct 30 01:37:57.911177 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:57.911144 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-ccb2j_04691eac-59b2-40f2-bea3-84cba2e43e17/istio-proxy/0.log" Oct 30 01:37:58.508930 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:58.508904 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kgxxw_1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0/istio-proxy/0.log" Oct 30 01:37:58.710590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:58.710563 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-lvppb_e2491bdb-6953-40e8-8c95-5c39bca71b56/istio-proxy/0.log" Oct 30 01:37:58.834084 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:37:58.834001 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:37:59.037415 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:59.037383 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz"] Oct 30 01:37:59.048121 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:59.048096 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-mq5dz"] Oct 30 01:37:59.106220 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:59.106184 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-srsm4_b20309a7-fa26-440e-80bb-955b9fac548b/istio-proxy/0.log" Oct 30 01:37:59.287902 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:59.287881 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qcfkv_97cce19f-6ecc-4cf9-8205-628cf08ec662/payload-pre-processing/0.log" Oct 30 01:37:59.874942 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:37:59.874907 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b76a46-a496-460c-be8e-0147ba6cfe01" path="/var/lib/kubelet/pods/11b76a46-a496-460c-be8e-0147ba6cfe01/volumes" Oct 30 01:38:00.437445 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:00.437417 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jrvcb_f8099822-8f87-43de-9860-0d34402a1669/main/0.log" Oct 30 01:38:00.623567 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:00.623536 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wpnlg_8a59a208-000a-448d-9ca1-6042908e69bb/main/0.log" Oct 30 01:38:01.008788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:01.008735 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d5llv_111fbfd3-f42e-4d29-b107-65ddfa99c0a4/main/0.log" Oct 30 01:38:01.385855 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:01.385832 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9bc8d_5826cc58-3128-42cc-b1e6-06847d5ed3b2/main/0.log" Oct 30 01:38:16.029418 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:16.029381 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc"] Oct 30 01:38:16.040490 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:16.040463 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ls2fc"] Oct 30 01:38:17.876052 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:17.876006 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31b2986-29c6-49a2-95db-40e51c5ce410" path="/var/lib/kubelet/pods/a31b2986-29c6-49a2-95db-40e51c5ce410/volumes" Oct 30 01:38:28.834517 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:38:28.834476 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:38:43.627053 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:38:43.627018 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:58.833497 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:38:58.833457 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:39:09.028621 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:09.028586 5938 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88"] Oct 30 01:39:09.039801 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:09.039773 5938 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1b4153b-9ts88"] Oct 30 01:39:09.873964 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:09.873933 5938 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62dedb8-7dc2-4e9d-ba83-09214ca25fa2" path="/var/lib/kubelet/pods/f62dedb8-7dc2-4e9d-ba83-09214ca25fa2/volumes" Oct 30 01:39:19.149178 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:19.149142 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:21.055175 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:21.055141 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:23.491436 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:23.491407 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h282r_9e691205-7020-4ddd-9fe6-61bb9d460d30/global-pull-secret-syncer/0.log" Oct 30 01:39:24.275307 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:24.275269 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" Oct 30 01:39:24.861247 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:24.861218 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-130.ec2.internal_87b9cc45a336a6ed23d0ff1b116a4584/haproxy/0.log" Oct 30 01:39:28.832979 ip-10-0-131-130 kubenswrapper[5938]: E1030 01:39:28.832939 5938 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mcpcw" containerName="extractor" Oct 30 01:39:31.153840 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:31.153807 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797fb9856f-8rq9w_cacb0b7a-9b96-4964-b8cc-cab7a5c17e43/authorino/0.log" Oct 30 01:39:32.022081 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:32.022047 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-n6n52_8098738c-5d92-4b27-9196-c0071d6590e4/manager/0.log" Oct 30 01:39:32.855817 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:32.855780 5938 scope.go:122] "RemoveContainer" containerID="c9769c7a3075baf3b01c9a27822a03636aa3770fc5a2ccfef3654f7fbe751323" Oct 30 01:39:32.869448 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:32.869428 5938 scope.go:122] "RemoveContainer" containerID="498e0452bb836cde26ed3324eecdb3f5a6a30da90696b57531b67af803c7f905" Oct 30 01:39:33.367002 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.366973 5938 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf"] Oct 30 01:39:33.367556 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.367538 5938 cpu_manager.go:414] "containerMap: removing container" podUID="97e75580-3fbc-42fc-a2f2-35af31faa68e" containerName="maas-api" Oct 30 01:39:33.367609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.367559 5938 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e75580-3fbc-42fc-a2f2-35af31faa68e" containerName="maas-api" Oct 30 01:39:33.367721 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.367706 5938 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e75580-3fbc-42fc-a2f2-35af31faa68e" containerName="maas-api" Oct 30 01:39:33.372754 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.372736 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.388928 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.388899 5938 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhfxw\"/\"default-dockercfg-tt658\"" Oct 30 01:39:33.389819 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.389796 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf"] Oct 30 01:39:33.399968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.399945 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"openshift-service-ca.crt\"" Oct 30 01:39:33.400522 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.400503 5938 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"kube-root-ca.crt\"" Oct 30 01:39:33.465920 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.465893 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-podres\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.466040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.465929 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-proc\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.466040 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.465963 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-sys\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.466138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.466090 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-lib-modules\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.466189 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.466125 5938 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv45v\" (UniqueName: \"kubernetes.io/projected/e43cf33a-202e-4487-bb34-f9643fabef50-kube-api-access-vv45v\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566790 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566762 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-lib-modules\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566798 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv45v\" (UniqueName: \"kubernetes.io/projected/e43cf33a-202e-4487-bb34-f9643fabef50-kube-api-access-vv45v\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566853 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-podres\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566903 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-proc\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566918 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-lib-modules\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.566939 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566936 5938 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-sys\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.567176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.566990 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-podres\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.567176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.567008 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-proc\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.567176 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.567046 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43cf33a-202e-4487-bb34-f9643fabef50-sys\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.590241 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.590219 5938 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv45v\" (UniqueName: \"kubernetes.io/projected/e43cf33a-202e-4487-bb34-f9643fabef50-kube-api-access-vv45v\") pod \"perf-node-gather-daemonset-hm5zf\" (UID: \"e43cf33a-202e-4487-bb34-f9643fabef50\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.685998 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.685937 5938 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:33.982647 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:33.982568 5938 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf"] Oct 30 01:39:33.990017 ip-10-0-131-130 kubenswrapper[5938]: W1030 01:39:33.989985 5938 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode43cf33a_202e_4487_bb34_f9643fabef50.slice/crio-7659ac88fa1b1391a3bb06a9d79f3f13659d79fe0f37ead5362af722e2258389 WatchSource:0}: Error finding container 7659ac88fa1b1391a3bb06a9d79f3f13659d79fe0f37ead5362af722e2258389: Status 404 returned error can't find the container with id 7659ac88fa1b1391a3bb06a9d79f3f13659d79fe0f37ead5362af722e2258389 Oct 30 01:39:34.050570 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:34.050541 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" event={"ID":"e43cf33a-202e-4487-bb34-f9643fabef50","Type":"ContainerStarted","Data":"7659ac88fa1b1391a3bb06a9d79f3f13659d79fe0f37ead5362af722e2258389"} Oct 30 01:39:34.642209 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:34.642180 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2jd9q_d4ed79ad-9a13-4899-8307-eff21d88d3e6/kube-state-metrics/0.log" Oct 30 01:39:34.756704 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:34.756678 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2jd9q_d4ed79ad-9a13-4899-8307-eff21d88d3e6/kube-rbac-proxy-main/0.log" Oct 30 01:39:34.873219 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:34.873197 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2jd9q_d4ed79ad-9a13-4899-8307-eff21d88d3e6/kube-rbac-proxy-self/0.log" Oct 30 01:39:34.994046 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:34.993988 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56f7b49654-f62f5_3757f105-2260-48d2-be57-41ba8481ae63/metrics-server/0.log" Oct 30 01:39:35.057092 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:35.057065 5938 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" event={"ID":"e43cf33a-202e-4487-bb34-f9643fabef50","Type":"ContainerStarted","Data":"8fbf9a3f0ef5d3b385231784b9c5789384992b1089acf2cd65c4af00bac86b8d"} Oct 30 01:39:35.057231 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:35.057206 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:35.096203 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:35.096155 5938 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" podStartSLOduration=2.096143721 podStartE2EDuration="2.096143721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:35.096089958 +0000 UTC m=+4027.336325072" watchObservedRunningTime="2026-10-30 01:39:35.096143721 +0000 UTC m=+4027.336378861" Oct 30 01:39:35.115134 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:35.115111 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-fqf4v_7876ee6c-178d-4ca6-923a-d76c0867bd85/monitoring-plugin/0.log" Oct 30 01:39:35.997746 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:35.997717 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzdmx_6283ace2-25a6-43fe-b12c-1d952c5c95da/node-exporter/0.log" Oct 30 01:39:36.114465 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.114441 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzdmx_6283ace2-25a6-43fe-b12c-1d952c5c95da/kube-rbac-proxy/0.log" Oct 30 01:39:36.232665 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.232645 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzdmx_6283ace2-25a6-43fe-b12c-1d952c5c95da/init-textfile/0.log" Oct 30 01:39:36.353854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.353835 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qvj6_84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/kube-rbac-proxy-main/0.log" Oct 30 01:39:36.470389 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.470361 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qvj6_84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/kube-rbac-proxy-self/0.log" Oct 30 01:39:36.587535 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.587502 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-4qvj6_84d34e3a-70d1-43c0-9a10-8f8be0b4dbfc/openshift-state-metrics/0.log" Oct 30 01:39:36.714916 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.714834 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/prometheus/0.log" Oct 30 01:39:36.830788 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.830764 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/config-reloader/0.log" Oct 30 01:39:36.948084 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:36.948059 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/thanos-sidecar/0.log" Oct 30 01:39:37.065816 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.065741 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/kube-rbac-proxy-web/0.log" Oct 30 01:39:37.183613 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.183577 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/kube-rbac-proxy/0.log" Oct 30 01:39:37.301105 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.301068 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/kube-rbac-proxy-thanos/0.log" Oct 30 01:39:37.418609 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.418590 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8233ecf8-dc61-4a5c-a234-bd021ba93fae/init-config-reloader/0.log" Oct 30 01:39:37.545629 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.545606 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r5ptl_c9149ff3-efaf-4600-b5fe-7699ffcd8db5/prometheus-operator/0.log" Oct 30 01:39:37.658775 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:37.658755 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r5ptl_c9149ff3-efaf-4600-b5fe-7699ffcd8db5/kube-rbac-proxy/0.log" Oct 30 01:39:41.071941 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:41.071917 5938 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-hm5zf" Oct 30 01:39:43.735014 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:43.734974 5938 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5zk8x_1e9caa4b-59b7-47c9-a7d5-0141bd0d25fc/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:46.552787 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:46.552756 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-846tj_08604272-a8a8-44c9-bcc1-9b3f12a37d77/dns/0.log" Oct 30 01:39:46.666551 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:46.666522 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-846tj_08604272-a8a8-44c9-bcc1-9b3f12a37d77/kube-rbac-proxy/0.log" Oct 30 01:39:47.591947 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:47.591921 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m75x5_34aaeeff-dda3-47f1-84cf-ea29bb1bd772/dns-node-resolver/0.log" Oct 30 01:39:48.890968 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:48.890941 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-w85dq_b2695552-6541-4a4b-90ab-c26a57b3c570/kube-auth-proxy/0.log" Oct 30 01:39:49.058138 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.058105 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5f7mc_56772ce8-a261-4951-b032-2b02e0f486ec/node-ca/0.log" Oct 30 01:39:49.208854 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.208769 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-ccb2j_04691eac-59b2-40f2-bea3-84cba2e43e17/istio-proxy/0.log" Oct 30 01:39:49.410423 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.410392 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-kgxxw_1f97dbd6-ff80-40ec-bfd5-b1ed49b70ab0/istio-proxy/0.log" Oct 30 01:39:49.489590 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.489514 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-lvppb_e2491bdb-6953-40e8-8c95-5c39bca71b56/istio-proxy/0.log" Oct 30 01:39:49.644381 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.644357 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-srsm4_b20309a7-fa26-440e-80bb-955b9fac548b/istio-proxy/0.log" Oct 30 01:39:49.715525 ip-10-0-131-130 kubenswrapper[5938]: I1030 01:39:49.715503 5938 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qcfkv_97cce19f-6ecc-4cf9-8205-628cf08ec662/payload-pre-processing/0.log"